All bookinfo reviews pods in CrashLoopBackOff state CentOS

@Vikash082 commented on Thu Apr 12 2018

Is this a BUG or FEATURE REQUEST?:  BUG
Did you review https://istio.io/help/ and existing issues to identify if this is already solved or being worked on?:
Yes
Bug:
Y
What Version of Istio and Kubernetes are you using, where did you get Istio from, Installation details
istioctl version: 0.7.1
kubectl version: v1.10.0
CentOS Linux release 7.4.1708

Is Istio Auth enabled or not ?
Did you install the stable istio.yaml, istio-auth.yaml....  or if using the Helm chart please provide full command line input.
kubectl apply -f install/kubernetes/istio-auth.yaml

What happened:
All the three reviews-v1-xxxx pods are crashing. Used kubectl logs  to get the below log.
CWWKE0005E: The runtime environment could not be launched.
CWWKE0018E: An exception occurred while launching the runtime environment: java.lang.IllegalStateException: The state  could not be created. state=/opt/ibm/wlp/output/defaultServer/logs/state
com.ibm.ws.kernel.boot.LaunchException: Caught unexpected exception java.lang.IllegalStateException: The state  could not be created. state=/opt/ibm/wlp/output/defaultServer/logs/state
        at com.ibm.ws.kernel.boot.internal.KernelBootstrap.rethrowException(KernelBootstrap.java:524)
        at com.ibm.ws.kernel.boot.internal.KernelBootstrap.go(KernelBootstrap.java:211)
        at com.ibm.ws.kernel.boot.Launcher.handleActions(Launcher.java:244)
        at com.ibm.ws.kernel.boot.Launcher.createPlatform(Launcher.java:119)
        at com.ibm.ws.kernel.boot.cmdline.EnvCheck.main(EnvCheck.java:59)
        at com.ibm.ws.kernel.boot.cmdline.EnvCheck.main(EnvCheck.java:35)
Caused by: java.lang.IllegalStateException: The state  could not be created. state=/opt/ibm/wlp/output/defaultServer/logs/state
        at com.ibm.ws.kernel.boot.internal.KernelUtils.cleanDirectory(KernelUtils.java:213)
        at com.ibm.ws.kernel.boot.internal.KernelBootstrap.clearServerStateDir(KernelBootstrap.java:226)
        at com.ibm.ws.kernel.boot.internal.KernelBootstrap.go(KernelBootstrap.java:102)
        ... 4 more

What you expected to happen:
All the three review pods should run.
How to reproduce it:
kubectl apply -f samples/bookinfo/kube/bookinfo.yaml

Feature Request:
N
Describe the feature:

@ayj commented on Thu Apr 12 2018
Can you provide more information on your test environment / cluster?
cc @linsun @frankbu - any ideas why the bookinfo review app might be crashing?

@Vikash082 commented on Thu Apr 12 2018
I have 2 physical nodes, one master and one worker. These two machines have 57G free memory and enough disk space. Both the nodes are Centos systems.
Pod info:
$ kubectl get pods --all-namespaces
NAMESPACE      NAME                                      READY     STATUS             RESTARTS   AGE
default        details-v1-7b97668445-4s9vl               1/1       Running            0          3h
default        productpage-v1-7bbdd59459-27ndc           1/1       Running            0          3h
default        ratings-v1-76dc7f6b9-qm5s4                1/1       Running            0          3h
default        reviews-v1-64545d97b4-4n246               0/1       CrashLoopBackOff   46         3h
default        reviews-v2-8cb9489c6-c9bdq                0/1       CrashLoopBackOff   46         3h
default        reviews-v3-6bc884b456-pw4t8               0/1       CrashLoopBackOff   46         3h
istio-system   istio-ca-75fb7dc8d5-xdtcc                 1/1       Running            0          6h
istio-system   istio-ingress-8546966f58-4c9r8            1/1       Running            0          6h
istio-system   istio-mixer-566f68f5d6-zkqg5              3/3       Running            0          6h
istio-system   istio-pilot-fd8fb6957-w6m4w               2/2       Running            0          6h
istio-system   istio-sidecar-injector-844b9d4f86-mgbjl   1/1       Running            0          3h
kube-system    etcd-os-ctrl-14                           1/1       Running            1          6d
kube-system    kube-apiserver-os-ctrl-14                 1/1       Running            1          6d
kube-system    kube-controller-manager-os-ctrl-14        1/1       Running            1          6d
kube-system    kube-dns-86f4d74b45-wwvwc                 3/3       Running            24         6d
kube-system    kube-flannel-ds-qtpz7                     1/1       Running            0          6d
kube-system    kube-flannel-ds-srrgc                     1/1       Running            1          6d
kube-system    kube-proxy-5pnmz                          1/1       Running            1          6d
kube-system    kube-proxy-p98rp                          1/1       Running            0          6d
kube-system    kube-scheduler-os-ctrl-14                 1/1       Running            1          6d

cluster-info:
 kubectl cluster-info
Kubernetes master is running at https://192.168.xx.yy:6443
KubeDNS is running at https://192.168.xx.yy:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Let me know, if more information is needed.

Maybe not enough resource on your minikube? I'm running bookinfo fine on my cluster, but i have mTLS disabled.
Is this only happening when mTLS is enabled?

@linsun This issue is seen with Centos based systems. I am not using minikube. I have 1 master and 1 worker node on physical servers. On both the setups there are ~57G memory available.
However, I also tried this on Ubuntu 16.04 and it worked fine on it, with and without TLS.

This doesn't seem like an Istio bug, but could boil down to how either Kubernetes or Docker were configured on CentOS. It isn't a lack-of-resources problem, because these pods are actually getting scheduled and executed (or else they wouldn't even have logs). The logs indicate that something is going wrong inside the container while bringing up the Liberty environment (the platform that the reviews service is built on).
Specifically, at start-up time, Liberty tries to clean up its state directory by first recursively deleting it, then recreating it (source code here). Even though this logic executes inside a container (and so hopefully it would have consistent behavior across all Docker-supported environments), I know that some of Java's behavior is still platform-specific, and file I/O is the kind of thing that still needs to be handled by the underlying Kernel, so it may just legitimately have slightly different behavior on different distributions.
Another theory is that, even though the default run-as user for this image is root, either the CentOS dockerd configuration or your Kubernetes configuration may have more security-conscious settings, and try to use a less privileged user by default? I don't know if either Docker or Kubernetes actually supports this kind of configuration, but if these containers aren't running as root, it would certainly explain having a permission error when trying to create this directory.
@Vikash082, if it's worth it for you to get to the bottom of this and root cause what's going wrong with CentOS, could you also provide information about how you set up both Kubernetes and Docker so that we could try to reproduce this problem? It would also be great if you were able to confirm if the reviews service is actually running as root in its container.

@pipejakob Here are the steps, I followed for installation:
- Disabled SELINUX on both the nodes.
- Installed docker, kubectl, kubeadm and kubelet as root.
- Initialize pod-networking using flannel.
- Created a non-root user on master node.
On worker node:
- Installed docker, kubectl, kubeadm and kubelet as root.
- Joined master node as root.

I am not able to determine the whether the reviews service is actually running as root or any other user as it is giving me below error:
error: unable to upgrade connection:
Unauthorized

I am running into the same issue with kubeadm & centos7, any idea to fix it?

I am running into the same issue with kubeadm & centos7, any idea to fix it?

I am running into the same issue with kubeadm & centos7, any idea to fix it?

I am running into the same issue with kubeadm & centos7. Then I rewrite the book-review service into spring boot, hope it can help. https://github.com/SpruceWang/istio-book-info

@pangsq thanks ,it work for me to add this:
env:
        - name: WLP_OUTPUT_DIR
          value: /tmp/output

@pangsq I have tried your method but it still does not work. May I ask did you change any other configurations?

we run into similar issue with kubelet v1.11.8+istio-1.1.4+docker 1.13.1, and adding "env"option does not work for us, we solved this issue after changing
/etc/sysconfig/docker-storage DOCKER_STORAGE_OPTIONS from overlay2 to devicemapper.
here is the final content:
DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper "
and you need stop kubelet/docker before changing this config, and restart docker/kubelet after changing this config.

we run into similar issue with kubelet v1.11.8+istio-1.1.4+docker 1.13.1, and adding "env"option does not work for us, we solved this issue after changing
/etc/sysconfig/docker-storage DOCKER_STORAGE_OPTIONS from overlay2 to devicemapper.
here is the final content:
DOCKER_STORAGE_OPTIONS="--storage-driver devicemapper "
and you need stop kubelet/docker before changing this config, and restart docker/kubelet after changing this config.

devicemapper is not ideal for a variety of reasons.

Regarding this issue, containers are not 100% portable between distros.  Istio's containers are built on Ubuntu, and some of the compiled against an Ubuntu libc (which really needs to match the kernel for full compatibility).
I am not a bookinfo wizard, but on inspection I'd take a look for
setcap
setpcap

in the dockerfiles.
The reason I'd suggest looking for these setcap/setpcap calls is that CentOS (atleast v 7) is incompatible with containers built against more modern kernels.  The common workaround used is to change to DM.
There are other various subtle incompatibilities (at the syscall layer between distros.  The best solution is to build per distro (or implement a distroless version of bookinfo).


ConnectionResetError(104, 'Connection reset by peer') causes re-download from scratch

First of all: great lib @iTaybb -- Thank you for putting it together.
I'm dealing with large files (~17Gb) being transferred from Azure CDN to Amazon EC2. The internet connection is stable I think but I'm noticing an issue that's causing a lot of pain for me:
When the host from which the files are getting downloaded resets connection, the SmartDL client just gives up and restarts the download from scratch. This of course is a big issue when I'm trying to get such large files. Below is sample log & code.
This is somewhat related to issue #19  but not entirely since Connection reset by peer is technically not lack of internet connection but rather a standard way HTTP endpoints communicate.
Seems to me SmartDL should be able to deal with connection reset and just ask to re-download the http chunk / range it was working on at that time. Thoughts?
CODE:
def download_url(self, url, location):
    downloader = SmartDL(url, location, progress_bar=False, fix_urls=False, threads=10, logger=logger)
    downloader.timeout = 20
    downloader.attemps_limit = 10
    downloader.minChunkFile = 64 * (1024 ** 2)  # 64 MB

    try:
        downloader.start(blocking=False)

        while not downloader.isFinished():
            logger.info('Upload progress: SIZE: {} {} SPEED: {} ETA: {}'.format(
                "{0:.1f}%".format(downloader.get_progress() * 100),
                downloader.get_dl_size(human=True),
                downloader.get_speed(human=True),
                downloader.get_eta(human=True),
            ), url=url, location=location, status=downloader.get_status())
            time.sleep(5)

        if downloader.isSuccessful():
            logger.info('Upload completed',
                        url=url,
                        location=downloader.get_dest(),
                        duration=downloader.get_dl_time(human=True))
        else:
            logger.error('Upload error',
                         url=url,
                         location=downloader.get_dest(),
                         errors=downloader.get_errors())

            errors_text = ''
            for e in downloader.get_errors():
                errors_text = errors_text + "\n" + str(e)

            raise Exception(errors_text)
    except Exception as error:
        file_list = glob.glob('{}*'.format(location))
        for file_path in file_list:
            try:
                os.remove(file_path)
            except:
                logger.error('Error while deleting file', file_path=file_path)
        raise error

LOG:
"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 88.0% 15.06 GB SPEED: 7.1 MB/s ETA: 4 minutes, 50 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:20.608056Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 88.2% 15.09 GB SPEED: 7.1 MB/s ETA: 4 minutes, 57 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:25.626230Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 88.4% 15.12 GB SPEED: 7.8 MB/s ETA: 4 minutes, 58 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:30.666378Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 88.6% 15.15 GB SPEED: 7.5 MB/s ETA: 4 minutes, 39 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:35.681668Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 88.8% 15.19 GB SPEED: 7.4 MB/s ETA: 4 minutes, 21 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:40.713161Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 89.0% 15.22 GB SPEED: 8.2 MB/s ETA: 4 minutes, 23 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:45.718700Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 89.2% 15.25 GB SPEED: 4.8 MB/s ETA: 5 minutes, 1 second", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:50.724313Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 89.3% 15.27 GB SPEED: 4.9 MB/s ETA: 6 minutes, 23 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:08:55.729961Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 89.5% 15.30 GB SPEED: 5.5 MB/s ETA: 5 minutes, 54 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:00.735431Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 89.6% 15.32 GB SPEED: 3.6 MB/s ETA: 7 minutes, 59 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:05.740615Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 89.7% 15.34 GB SPEED: 3.9 MB/s ETA: 7 minutes, 56 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:10.774994Z"}
{"event": "ConnectionResetError(104, 'Connection reset by peer')", "level": "exception", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.287198Z"}
NoneType
{"event": "Starting a new SmartDL operation.", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.287604Z"}
{"event": "One URL is loaded.", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.287741Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.287870Z"}
{"event": "Content-Length is<PHONE_NUMBER>0 (17.10 GB).", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.615813Z"}
{"event": "Launching 10 threads (downloads 1.71 GB/thread).", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.616141Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.000'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.617045Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.001'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.617507Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.002'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.617891Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.003'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.618336Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.004'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.618734Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.005'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.619121Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.006'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.657019Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.007'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.657721Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.008'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.658263Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b.009'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.659037Z"}
{"event": "Control thread has been started.", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.676908Z"}
{"event": "Diff between downloaded files and expected filesizes is<PHONE_NUMBER>.0kB.", "level": "warning", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.677279Z"}
{"event": "Starting a new SmartDL operation.", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.677813Z"}
{"event": "One URL is loaded.", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.677947Z"}
{"event": "Downloading 'http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014' to '/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b'...", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.678064Z"}
{"event": "Content-Length is<PHONE_NUMBER>0 (17.10 GB).", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.760694Z"}
{"event": "Launching 10 threads (downloads 1.71 GB/thread).", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:12.761018Z"}
{"event": "Control thread has been started.", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:13.558547Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 0.2% 27.2 MB SPEED: 17.0 MB/s ETA: 0 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:15.931493Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 0.6% 108.5 MB SPEED: 21.1 MB/s ETA: 14 minutes, 36 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:20.949592Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 1.1% 197.7 MB SPEED: 20.5 MB/s ETA: 14 minutes, 11 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:25.958432Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 1.6% 286.0 MB SPEED: 19.7 MB/s ETA: 14 minutes, 22 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:30.994674Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 2.1% 373.6 MB SPEED: 21.1 MB/s ETA: 14 minutes, 44 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:36.071302Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 2.7% 464.9 MB SPEED: 20.4 MB/s ETA: 14 minutes, 2 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:41.314663Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 3.1% 548.0 MB SPEED: 19.3 MB/s ETA: 15 minutes, 29 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:46.325928Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 3.6% 631.5 MB SPEED: 18.4 MB/s ETA: 15 minutes, 5 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:51.337558Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 4.1% 718.0 MB SPEED: 20.0 MB/s ETA: 14 minutes, 28 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:09:56.537931Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 4.6% 808.3 MB SPEED: 21.2 MB/s ETA: 13 minutes, 51 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:10:01.544831Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 5.1% 892.5 MB SPEED: 19.2 MB/s ETA: 14 minutes, 2 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014", "level": "info", "logger": "content.the_client", "timestamp": "2019-06-22T19:10:06.578321Z"}
{"location": "/tmp/3099cbe6-3496-4e64-9d9e-4aea8a90fe4b", "event": "Upload progress: SIZE: 6.0% 1.03 GB SPEED: 36.9 MB/s ETA: 8 minutes, 57 seconds", "status": "downloading", "url": "http://HOST-OBFUSCATED/2018/ACC/2e9046c1564bb85f46fd97d3ab2c07f6adcbb508.mp4?sv=2014-02-14&sr=b&sig=6iaaT3ZKnuLUbfgDyBi57i0p8AR6el3e%2BKhEM0nw1uI%3D&se=2019-06

I believe this wraps up #19 and #14.
I've already started to implement it on a side branch (resume-stopped-downloads), but it's not working yet.
I hope i'll have time in the next few days to address this feature.

Awesome - thank you.  I checked out the resume-stopped-downloads but noticed that it it creates really big part files .. I suspect it doesn't currently calculate from which point the download needs to place based on existing file size. Good luck with that and I'll watch the repo. Thank you again!!


DropDownMenu selectedIndex issue when index > options length

We're getting an error when the selectedIndex is greater than the length of the options.
We're even getting an error when the index is 0 and no options have been set yet. This is an issue when getting options from the server. This throws an error:
{this.props.menuItems[this.state.selectedIndex].text}
It seems like any index should be ignored if there are no options and indexes >= length should be ignored as well.
What do you think? I can fix this if there's no objection.

+1

Sounds reasonable to me - Thanks @FOr3st

the fixes we're doing for #51 should cover this.  we encountered something similar when menuItems = [].

Great, thanks @chrismcv

#846 should close this

Thanks @chrismcv

Thanks @chrismcv


cuda/ViterbiPath.cuh: No such file or directory with cmake 3.5.1

Bug Description
Compilation error with ViterbiPath.cu
flashlight/lib/sequence/criterion/cuda/ViterbiPath.cu:8: error: flashlight/lib/sequence/criterion/cuda/ViterbiPath.cuh: No such file or directory
Reproduction Steps
git clone (todays master)
Using cmake version 3.5.1:
cmake -DFL_BACKEND=CUDA -DFL_LIBRARIES_USE_CUDA=ON  -DCMAKE_VERBOSE_MAKEFILE=1
make
see compilation failure
Platform and Hardware
ubuntu 16.04, cuda 10.2, gcc5.4
Additional Context
This is due to
cuda_include_directories(
${CUB_INCLUDE_DIRS}
$<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/..> # for #include "flashlight/lib"
)
in repos/flashlight/lib/CMakeLists.txt
which is not resolve as expected :
/usr/local/cuda/bin/nvcc -M -D__CUDACC__ flashlight/lib/sequence/criterion/cuda/ViterbiPath.cu -o flashlight/debug/CMakeFiles/fl-libraries-cuda.dir/lib/sequence/criterion/cuda/fl-libraries-cuda_generated_ViterbiPath.cu.o.NVCC-depend -ccbin /usr/bin/cc -m64 -Xcompiler ,"-g" -std=c++11 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_70,code=sm_70 -gencode arch=compute_75,code=sm_75 -gencode arch=compute_70,code=compute_70 -gencode arch=compute_75,code=compute_75 -DNVCC -Iflashlight/debug/CUB/src/CUB -I$<BUILD_INTERFACE:flashlight/..> -I/usr/local/cuda/include -I/usr/local/cuda/include
See: -I$<BUILD_INTERFACE:flashlight/..>
Seems to be fixable with simply:
cuda_include_directories(
${CUB_INCLUDE_DIRS}
${PROJECT_SOURCE_DIR}/..
)

@WilliamTambellini — bleh, sorry about this. I've seen this before a few times, but it's hard to repro. Your fix will fix the issue for make - the issue is that make install won't work properly in that case (you can try it) because the include path will have a prefix which shouldn't be installed.
Can you run make install after building with a clean version of the repo and let me know what happens?

@jacobkahn  Well I could not simply build because both make and make install failed.
Nevermind, it seems to be a cmake thing (bug?) : fails with 3.5.1 but works with 3.13.
The official cmake doc about INTERFACE_INCLUDE_DIRECTORIES
https://cmake.org/cmake/help/v3.5/prop_tgt/INTERFACE_INCLUDE_DIRECTORIES.html
actually points to 3.5.2, not 3.5.1. Perhaps they fixed a bug in 3.5.2 but FL cmake still accepts 3.5.1 as minimum. CU.

Closing, as 3.10 is now the minimum required CMake version.


In IvorySQL the function (to_date\to_timestamp\to_timestamp_tz) result is different from the Oracle result

Bug Report
When the first parameter of the function (to_date\to_timestamp\to_timestamp_tz) is passed only the value of the year, the converted result is different from the Oracle result.
IvorySQL Version
1.1 and 1.2
OS Version (uname -a)
Linux
Configuration options  ( config.status --config )
compatible_mode =oracle
Current Behavior
When the first parameter of the function (to_date\to_timestamp\to_timestamp_tz) is passed only the value of the year, the month part in IvorySQL is January, while the month part in Oracle is February (the current month).
Test SQL：
System time：2022/02/28
select to_date('2020','yyyy'),to_timestamp('2020','yyyy'),to_timestamp_tz('2020','yyyy') from dual;
Test result：
IvorySQL：The year, month and day are 2020-01-01

Oracle：The year, month and day are 2020-02-01

Expected behavior
Please confirm the result of IvorySQL and Oracle, which one is correct.

Well, there are differences, so please work with us to figure out why Orcale is doing this, and if we make it January, will it have a fatal impact on the application? The growth of the community needs us to work together!

@BingAo-Office , example
System time：2022-02-03
IvorySQL's result：2020-01-01
Oracle's result：2020-02-01
System time：2022-03-03
IvorSQL's result：2020-01-01
Oracle's result：2020-03-01
I.e., IvorySQL always gets the 1st of January. But Oracle gets the 1st day of the current month in system time.


Upgrade to Python 3.12

Upgrading to python 3.12 provides a ton of benefits both in authoring and in runtime performance. Of particular note is the new type param syntax, which will massively improve the readability of the relatively large amount of generic typing that we make use of. In addition, there's huge performance gains in asyncio as a whole.
Note that this PR doesn't actually update any of the python code to use the new syntax - that can come as a separate PR. It's also likely that we'll need to update mypy (or move to ruff) to be able to use the new syntax.
To facilitate this upgrade, pants had to also be upgraded to 2.19. Remarkably, it wasn't a whole day affair to do. The big change that had to be made was to add everything to the lockfiles so that they wouldn't try to pull some pre-compiled pex that doesn't support 3.12
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 85.09%. Comparing base (23f664c) to head (b361797).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff            @@
##           develop     #198   +/-   ##
========================================
  Coverage    85.09%   85.09%
========================================
  Files           37       37
  Lines         1134     1134
========================================
  Hits           965      965
  Misses         169      169

Flag
Coverage Δ

unittests
85.09% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


ValueError: Unknown layer: Merge

Sorry to disturb you  when i trying to use your model  i found this error as the title, i guess it may be cause of the version of keras ,whats your opinion  thanks

Hi, since I droped out my PhD education, I deleted all content about this project from my computer except this repo unfortunately. Because of that, I cannot help you. Sorry


swiper for dashboard

The Dashboard now uses the new Swiper Component. I had to deactivate the looping through the swiper items, because the swiper library generates the swiper items on its own and the follow listeners are only registered to the initial items. This has the consequence that the follow sometimes works and sometimes not.
The Follow icon now overlays the entire artist image. For round images, positioning in the upper right corner no longer works so well.

I also removed the date tooltips. That became too fiddly for me. I would have also liked to use the new Bootstrap tooltips, but there were problems with other JavaScipt on the page. Would then address the issue again during the migration.

 New Swiper for Dashboard


Security vulnerability responsible disclosure

I am trying to communicate a discovered vulnerability in Databasir through the email address associated with @vran-dev, without success.
Can you share a way to report these findings or create a security advisory?
Thanks in advance

@ThomasTNO
thanks for your finding and sharing, maybe i was missing the email, it's so sorry.
back to the topic, first you may need to check if it is repeated as  Databasir SSR, if not i will create a security advisory.

Thanks for the quick response. It is different to Databasir SSR and still persists in the latest release

Thanks for the quick response. It is different to Databasir SSR and still persists in the latest release

the advisory had been created,  and has invite you as collaborators, next step need your help to wirte details about this vulnerability.
then i will review it and request CVE, thanks for your contribution

link here: https://github.com/vran-dev/databasir/security/advisories/GHSA-qvg8-427f-852q


Add Clubstep x Carly Ray Jepsen by DJ-Nate

Song Name
Clubstep x Carly Ray Jepsen
Artist Name
DJ-Nate
Source
Youtube
Youtube Link
9KFmJ25_5h8
Direct File Link
https://youtu.be/9KFmJ25_5h8?feature=shared
Song ID
13
Start Offset [ms]
0
Extra
Created from Auto Nong.
Created for level: 14
Downloaded: Yes

Accept
---------- Forwarded message ---------
De: auto-nong[bot] @.>
Date: dom, 27 de oct de 2024, 2:36 p. m.
Subject: Re: [FlafyDev/auto-nong-indexes] Add Clubstep x Carly Ray Jepsen
by DJ-Nate (Issue #5507)
To: FlafyDev/auto-nong-indexes @.>
Cc: Lux24GD @.>, Author @.>
Thanks for contributing!
Someone will soon review your request.
Here is the JSON result for the reviewer:
{
"name": "Clubstep x Carly Ray Jepsen",
"artist": "DJ-Nate",
"source": "youtube",
"startOffset": 0,
"yt-id": "9KFmJ25_5h8",
"songs": [
13
]
}
To accept the request, a reviewer needs to write "accept".
—
Reply to this email directly, view it on GitHub
https://github.com/FlafyDev/auto-nong-indexes/issues/5507#issuecomment-2440149156,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BMO6BS4KGT55ADTCH6JMVC3Z5U6E3AVCNFSM6AAAAABQWDKDECVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBQGE2DSMJVGY
.
You are receiving this because you authored the thread.Message ID:
@.***>


Error responses should throw?

I would expect the promise to get rejected, but it doesn't. Either way, the response status isn't available to the caller, as far as I can tell.
Will you take a PR?

Actually, MDN sez:

The Promise returned from fetch() won’t reject on HTTP error status even if the response is an HTTP 404 or 500. Instead, it will resolve normally (with ok status set to false), and it will only reject on network failure or if anything prevented the request from completing.

So maybe the solution is to just add a status code accessor?

Oh yeah, I forgot about this. I've been unsafely coercing and grabbing the status code in some test code at work.

Yep, that's what I'm doing too.


BZ-1248020 JobExecutor: Job Service Settings dialog active despite di…

…sabled job executor

merged in master.. more checking are coming in a separate PR


Improve perf

Codecov Report

Merging #3 (4bf9abe) into main (91b8684) will decrease coverage by 0.14%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##              main       #3      +/-   ##
===========================================
- Coverage   100.00%   99.85%   -0.15%
===========================================
  Files            4        4
  Lines          682      689       +7
===========================================
+ Hits           682      688       +6
- Misses           0        1       +1

Impacted Files
Coverage Δ

src/macros/int.rs
99.66% <100.00%> (-0.34%)
:arrow_down:

src/macros/uint.rs
100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Support tearDown in scheduled_test

 Issue by nex3
Originally opened as dart-lang/sdk#15689

When scheduled_test was first written, it omitted [tearDown] on the theory that the onComplete queue served the same purpose. However, since then it's become clear that it's occasionally useful to add additional [schedule] calls after each test case without waiting for the existing scheduled calls to complete, so [tearDown] should be added.

 Comment by nex3

Added Started label.

 Comment by anders-sandholm

Removed Library-ScheduledTest label.
Added Pkg-ScheduledTest label.

 Comment by nex3

Fixed in dart-lang/sdk@181a9358862894610b841107f7e31e46df2b43d8, published as scheduled_test 0.11.1.

Added Fixed label.


Video stop playing with error 'Cannot call method 'send' of null'

Hi,
I was playing a video , when suddenly it stopped with this error :
/usr/local/lib/node_modules/castnow/node_modules/castv2-client/node_modules/castv2/lib/client.js:126
this.ps.send(buf);
^
TypeError: Cannot call method 'send' of null
at Client.send (/usr/local/lib/node_modules/castnow/node_modules/castv2-client/node_modules/castv2/lib/client.js:126:11)
at Channel.send (/usr/local/lib/node_modules/castnow/node_modules/castv2-client/node_modules/castv2/lib/channel.js:34:12)
at RequestResponseController.Controller.send (/usr/local/lib/node_modules/castnow/node_modules/castv2-client/lib/controllers/controller.js:28:16)
at RequestResponseController.request (/usr/local/lib/node_modules/castnow/node_modules/castv2-client/lib/controllers/request-response.js:34:8)
at /usr/local/lib/node_modules/castnow/node_modules/chromecast-player/api.js:108:17
at Api.getCurrentSession (/usr/local/lib/node_modules/castnow/node_modules/chromecast-player/api.js:95:35)
at Api.sessionRequest (/usr/local/lib/node_modules/castnow/node_modules/chromecast-player/api.js:105:8)
at Api.pause (/usr/local/lib/node_modules/castnow/node_modules/chromecast-player/api.js:128:8)
at Object.ke%

This happens if castnow looses the connection to chromecast. I'm working on handleing this case better.


TB-3620 Adjust bottom sheet bottom padding depending upon device padding

What 🕵 🔍

What does the pull request solve?
What exactly was added or modified?

How to test 🥼 🔬

A brief description of the steps required for proper testing

Screenshots 📸 📱

Before
After

References 📝 🔗

source name

Codecov Report

Merging #113 (10a6dc3) into main (04a1b07) will not change coverage.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##             main     #113   +/-   ##
=======================================
  Coverage   47.41%   47.41%
=======================================
  Files          56       56
  Lines        2124     2124
=======================================
  Hits         1007     1007
  Misses       1117     1117

Impacted Files
Coverage Δ

lib/src/widget/bottom_sheet/bottom_sheet_base.dart
0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 04a1b07...10a6dc3. Read the comment docs.


Docs have incorrect restriction on TM_PROFILE-type Profile names

This Bug Report affects these Traffic Control components:

Documentation

Current behavior:
The docs for what Profile Types require what Profile Names (sort of) says that TM_PROFILEs can have a RASCAL or TM prefix - but *only Profiles with the prefix RASCAL are considered in Monitoring Snapshot generation.
Expected behavior:
The docs should be accurate.
Steps to reproduce:
Read the docs

Hello @ocket8888 So only TM part should be removed from the TM_PROFILEs ?

Yeah. There may be partial support for TM, but the docs shouldn't recommend it, because it won't work fully.


Migrate dense_image_warp op

Address #25. CI test passed on my local machine. Note that the test of _interpolate_bilinear in dense_image_warp will fail if I use the decorator tf.function on it (that's why I haven't added it yet). I'm still trying to figure out why.

Forget to create a new branch. Recreate a PR later.


Proposal: New way of rendering rich-text

Theme:

Improve rich-text rendering for .NET

Goals:

improve and unify the link & asset resolution (#258 - resolution of all data-* attributes, #243 - other attributes of , #193 - asset URLs)
promote ASP.NET display templates to a best practice
strengthen the object model (IRichTextContent was an early attempt to achieve that but it's far from complete)
separate responsibilities and clean up the code (e.g. bug #257 - resolving rich-text into different types, #231 - mixing rendering and resolution approaches)

All related issues: https://github.com/Kentico/kontent-delivery-sdk-net/labels/rich-text
Supported scenarios:

ASP.NET Display Templates
Flattening of rich-text for search indexing purposes, etc.
Provide ways to adjust content before rendering

Current solution:

We offer two types of rich-text resolution:

string-based - involves multiple types of resolvers for links, inline items, etc.
IRichTextContent-based - an object-like approach that combines hierarchical approach (for inline content items and assets) and string-based HTML rendering (for links)

Proposed solution:

Let the IRichTextContent be the default type for rich-text elements and provide an additional layer for flattening it to a string.

Detailed proposition:

Make the logic of RichTextContentConverter part of the ModelProvider
Adjust the logic of RichTextContentConverter to treat links as objects (=do not apply string-based resolution) => links, assets, inline content items will be treated the same - as objects
Provide resolution methods for all three type of objects ContentLinkResolver, ContentAssetResolver, InlineContentItemResolver

public string ResolveToString(T model) -> ability to flatten the object to string
public T Resolve(T model) -> ability to adjust model's properties

Enrich the existing models for Links and Asset with a collection for supplying additional attributes (such as class or target), also add properties for some of the most common attributes (such as URL)
Move all string-based resolution from the ModelProvider to a separate class (e.g. HtmlResolver)
Remove the --structuredmodel option from the code generator and leave it true by default

Example code:
var articleResponse = await deliveryClient.GetItemAsync<Article>("on-roasts");
Article article = articleResponse.Item;
IRichTextContent richText = article.Body;
// Contains links, assets, strongly typed inline content items...already processed by `T Resolve(T model)`
// Ready to be passed to ASP.NET MVC Display Templates
IEnumerable<IRichTextBlock> blocks = richText.Blocks;

HtmlResolver resolver = new HtmlResolver(new XYZLinkResolver(), new ABCAssetResolver()...); // Or build via DI
string flattenedHtml = resolver.ToString(blocks, articleResponse.LinkedItems);

It looks like this new approach might resolve this as the IContentLinkUrlResolver moves from ModelProvider to HtmlResolver,
But currently, because the IContentLinkUrlResolver is instantiated via DI in the ModelProvider, which is in turn used by the DeliveryClient, Its currently impossible to have a IDeliveryClient instance injected within a custom IContentLinkUrlResolver implementation (it creates a circular dependency), eg;
    public class CustomContentLinkUrlResolver : IContentLinkUrlResolver
    {
        public CustomContentLinkUrlResolver(IDeliveryClient client)
        {
        }
    }

which in our case we want, (and I would think is a pretty general requirement?) to query Kontent's subpages hierarchy and build the url for a particular content item passed to the Url Resolver.

Thanks for pointing this out, we will take this into consideration.


bug fix

This pull request resolves #162.
The pull request is ready to be merged.
The details of the pull request:

Pull Request Test Coverage Report for Build 589

9 of 9 (100.0%)  changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.007%) to 96.702%

Totals

Change from base Build 585:
0.007%

Covered Lines:
1730

Relevant Lines:
1789

💛  - Coveralls


get_certificate_name_string results in error: 'Name' object is not iterable

Describe what you wanted to do

is this error specific to a single file?

Yes, the androsign.py script that comes with androguard.

if possible give a minimal working example where the error happens

python androsign.py  -s
Describe what you expected
Certificate details.
Describe what actually happened
$ python3 androsign.py com.snapchat.android.apk -s
com.snapchat.android.apk, package: 'com.snapchat.android'
Is signed v1: True
Is signed v2: False
Found 1 unique certificates
Error in com.snapchat.android.apk
Traceback (most recent call last):
  File "androsign.py", line 57, in main
    print("Issuer:", get_certificate_name_string(x509_cert.issuer, short=True))
  File "/home/tapioca/.local/lib/python3.6/site-packages/androguard/util.py", line 49, in get_certificate_name_string
    return ", ".join(["{}={}".format(_.get(attr,(attr,attr))[0 if short else 1], name[attr]) for attr in name])
TypeError: 'Name' object is not iterable

System Information

Androguard Version: master
Python Version: 3.6.5
Operating System: Ubuntu 18.04.  Same behavior on Windows 10

Further Log Files and Output
I'm trying to update a script that was written for an older androguard version.  Based on the release notes at https://github.com/androguard/androguard/releases/tag/v3.2.0 , androsign.py should be referred to as new syntax for working with asn1crypto.  However, the script androsign.py itself doesn't appear to work.

asn1crypto 0.24.0 is installed on both machines, if it matters.

Can you upload the APK file here?

The error occurs with every APK I've tried.  So pick whichever you like.

this is really weird... I tested here with 0237c3848e1ebb204e08c9223c72c8092d51272e on Windows 7, Python 3.6.4 :: Anaconda, Inc., asn1crypto (0.24.0) and I get this result:
$ python androsign.py -s D:\com.snapchat.android.apk
com.snapchat.android.apk, package: 'com.snapchat.android'
Is signed v1: True
Is signed v2: False
Found 1 unique certificates
Issuer: C=01, ST=California, L=Los Angeles, O=Snapchat, OU=Captain Captcha, CN=Daniel Smith
Subject: C=01, ST=California, L=Los Angeles, O=Snapchat, OU=Captain Captcha, CN=Daniel Smith
Serial Number: 0x504f0d1c
Hash Algorithm: sha1
Signature Algorithm: rsassa_pkcs1v15
Valid not before: 2012-09-11 10:06:20+00:00
Valid not after: 2040-01-28 10:06:20+00:00
sha1 49f6badb81d89a9e38d65de76f09355071bd67e7

I also tested on a fresh virtualenv with debian buster, androguard 3.2.0 release,
there it looks like this:
$ ./foobar/bin/androsign.py -s com.snapchat.android.apk
com.snapchat.android.apk, package: 'com.snapchat.android'
Is signed v1: True
Is signed v2: False
Found 1 unique certificates
Issuer: Common Name: Daniel Smith, Organizational Unit: Captain Captcha, Organization: Snapchat, Locality: Los Angeles, State/Province: California, Country: 01
Subject: Common Name: Daniel Smith, Organizational Unit: Captain Captcha, Organization: Snapchat, Locality: Los Angeles, State/Province: California, Country: 01
Serial Number: 0x504f0d1c
Hash Algorithm: sha1
Signature Algorithm: rsassa_pkcs1v15
Valid not before: 2012-09-11 10:06:20+00:00
Valid not after: 2040-01-28 10:06:20+00:00
sha1 49f6badb81d89a9e38d65de76f09355071bd67e7

Can you try the following please:
cd /some/tmp/folder
virtualenv -p python3 androguard_venv
source ./androguard_venv/bin/activate
pip install androguard
./androguard_venv/bin/androsign.py -s com.snapchat.android.apk

maybe this is caused by some partially installed python stuff... On my machine it works fine in a fresh virtualenv, so I suspect it should work for you too.

The pip-installed version of androguard works.  But the current git version does not.  If I compare the two, I see:
@@ -53,8 +54,8 @@
             for cert in certs:
                 if args.show:
                     x509_cert = x509.Certificate.load(cert)
-                    print("Issuer:", x509_cert.issuer.human_friendly)
-                    print("Subject:", x509_cert.subject.human_friendly)
+                    print("Issuer:", get_certificate_name_string(x509_cert.issuer, short=True))
+                    print("Subject:", get_certificate_name_string(x509_cert.subject, short=True))
                     print("Serial Number:", hex(x509_cert.serial_number))
                     print("Hash Algorithm:", x509_cert.hash_algo)
                     print("Signature Algorithm:", x509_cert.signature_algo)
@@ -76,4 +77,4 @@

The androguard 3.2.0 release notes points to https://github.com/androguard/androguard/blob/master/androsign.py#L55 as an example of how you get certificate details.   But that file does not appear to have this change:
https://github.com/androguard/androguard/commit/b348ff61d844a5fbae3d411b14aa77f9297e6a44#diff-1926ea27a658230f583479928ff5c8a0
If you attempt to use the androsign.py file that uses the new syntax that appears to be recommended, it doesn't seem to work.

Since the change that I linked to was made a day after the 3.2.0 release, maybe this is as simple as pointing to the 3.2.0-release version of androsign.py, rather than master?

okay I see what the confusion is! The version of master needs to be used with the same version of https://github.com/androguard/androguard/blob/master/androguard/util.py
So If you are using master, right now, use the certficate processing as described in master.
Between 3.2.0 and master was a small change in the get_certificate_name_string function...
The link in the release description always points to master, which is wrong... I'll release a new version in the next days where all these bug fixes are included and fix the description then.

No, wait. You say that using 3.2.0 works but as soon as you use the master it does not? I need to set up the same environment as you have and test that...

okay I think I see the error... you say you are using master, but the error message from your first post tells me, that utils.py is not from master... it is the old version. So you have somehow the old version installed but using master for the script...

Right.  The situation is this:

I have androguard installed via pip.  This gets me 3.2.0
Looking at the androguard 3.2.0 release notes, it says "Please consult the API of asn1crypto or androsign.py how to access certain information of the certificate!"
When I clicked the link to androsign.py, I see code that doesn't work with 3.2.0

I added now some documentation for the certificate processing, which will be merged soon


Wrong judgement

Challenge Waypoint: Nest an Anchor Element within a Paragraph has an issue.
User Agent is: Mozilla/5.0 (X11; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0.
Please describe how to reproduce this issue, and include links to screenshots if possible.
My code:
<link href="http://fonts.googleapis.com/css?family=Lobster" rel="stylesheet" type="text/css">

CatPhotoApp

View more  cat photos

cat photos

Kitty ipsum dolor sit amet, shed everywhere shed everywhere stretching attack your ankles chase the red dot, hairball run catnip eat the grass sniff.

Purr jump eat the grass rip the couch scratched sunbathe, shed everywhere rip the couch sleep in the sink fluffy fur catnip scratched.

In the code above when I wrote the line: " View more  cat photos " at the end.It did not made all the problems correct. But when I used the same code just above the cat picture it showed correct output.
Attaching the screen shot for wrong output(line 38):

Attaching the screen shot for correct output(line 30):

There's no difference in the code except the position(line 38 in 1st screen shot vs line 30 in 2nd screen shot).

This test expects the wrapping p tag you insert to be the first p tag on the page. This had been fixed by a previous PR that has been overwritten. Someone will need to make a new PR that either makes it clear where the user needs to put their code or makes the test not care.

Yes,that should be made clear because otherwise wrong result for this is frustrating.


[grpc] Update to 1.28.0 and add feature absl-sync

This PR will:

Update to 1.28.0 release
Add feature absl-sync to use abseil synchronization module.

Related: #10242.
I will continue this PR after 1.28.0 release published.

Looks good so far! How far away is grpc 1.28.0? If it's far enough away, it might make sense to update the current version, and update to grpc 1.28.0 once that happens.

@strega-nil I don't know, maybe just recently.
However, in order to ensure the correctness of the code, I recommend updating after the official release version is released.

Is there any update for 1.28.(0/1) support. We have started using gRPC recently so don't currently have a production release, so we're hand-cranking this.
We would like to take advantage of the gRPC client retry policy which doesn't work in earlier releases before 1.27.3 due to json parser issues. 1.27.3 requires a code change to work and as we try and build vcpkg form source via a Jenkins job we cannot automate this.

Waiting for #11459.

/azp run

Thanks @JackBoosY :)


Best uniqueness locking for this gem

Hi, what is the best strategy for this gem? There are several options for https://github.com/mhenrixon/sidekiq-unique-jobs

We use the default uniqueness strategy.
Inside of our relevant workers we use:
sidekiq_options unique: true


Integrate track selection with the control center

As an AirPlay user I want to be able to change tracks directly from my locked device when using AirPlay. See WWDC 2019 for more information.
Acceptance criteria

The described feature is available in the Pillarbox demo.

Tasks

[ ] Check whether this feature is still working in recent versions of iOS (not sure). Check on iPhone and iPad, in the control center as well as on the lock screen.
[ ] Implement the feature if this is the case.

The API we were using in Letterbox was enableLanguageOptionCommand. I can confirm that the corresponding button is displayed in the iOS 15.7.2 control center when using AirPlay.


unable to use declarative-linter option with jenkins/jenkinsfile-runner image

When i try to use the jenkins/jenkinsfile-runner image to validate a Jenkinsfile in cli mode as follows
docker run --rm -i -v $(pwd)/Jenkinsfile:/workspace/Jenkinsfile jenkins/jenkinsfile-runner --cli

I either dont get any output in the console if i enter declarative-linter
WARNING: Using the deprecated CLI mode. Use the 'cli' subcommand instead of passing the argument or an environment variable
Connected to Jenkins!
Type 'help' for a list of available commands, or 'exit' to quit.
 > declarative-linter

or if i use the format declarative-linter < /workspace/Jenkinsfile , it throws an error
WARNING: Using the deprecated CLI mode. Use the 'cli' subcommand instead of passing the argument or an environment variable
Connected to Jenkins!
Type 'help' for a list of available commands, or 'exit' to quit.
 > declarative-linter <  /workspace/Jenkinsfile

ERROR: No argument is allowed: <
java -jar jenkins-cli.jar declarative-linter
Validate a Jenkinsfile containing a Declarative Pipeline

Can you tell me the correct way to use the declarative-linter option or is this a bug ?
Thanks

https://github.com/jenkinsci/jenkinsfile-runner/issues/225 needs to be implemented to support such mode in JFR

https://github.com/jenkinsci/jenkinsfile-runner/issues/225 needs to be implemented to support such mode in JFR

Hi @oleg-nenashev , so can we not use just the declarative-linter option ,ie, not specifying the the file like <  /workspace/Jenkinsfile  ,assuming that it is aware of the file in /workspace/Jenkinsfile  ?

Hi @oleg-nenashev , so can we not use just the declarative-linter option ,ie, not specifying the the file like <  /workspace/Jenkinsfile  ,assuming that it is aware of the file in /workspace/Jenkinsfile  ?


Not able to convert nested postman collection

For simple postman collections I am able to convert to OpenAPI. But when the postman collection JSON is nested, meaning it contains  "item" key inside "item" key, then I am getting the following error
Error: SyntaxError: Unexpected token u in JSON at position 0
I am using the library in the following way
   const result = postmanToOpenApi(str, null, {})

Hi @imranshaik012  sorry I don't know what is a nested postman collection, could you attach an example collection to inspect?
Thanks

Close due to inactivity

Hi @HeathNaylor, ok I understand the problem now, the library only supports one level of folders in postman, that's because in Open API there are no levels, you just can have tags that identify one level group of request, this is why don´t have sense to have more levels in a Postman collection that you want to transform to an Open API spec because there are no a possible "translation" for that. The folder support in the library is for the case where you want to group your request by resource type, entity... that is exactly the use case for the Open API tags.

@joolfe, well, can we at least have an error message for that? Or fix it by either converting from folders to tags or ignoring the folders entirely?

hi @AlexxNica ,
Since version 1.7.0 nested folder are supported, see docs for more info about the options and tests for examples.
Best Regards.


Improve overal structure & flow

[ ] How is the reading flow and how can we improve it?
[ ] How is the content structured? does it make sense? can we make it better?

Overview page:
please open this pdf file I created:
Overview page report.pdf

Get Started page:
The text could be improved there as well

Step 1 - Setup
The text could be improved to something like:
Manage all aspects of your Unbody account effortlessly through the comprehensive dashboard. From account settings and subscription management, projects and more, the dashboard provides you with complete control and flexibility in one seamless platform.

Step 2 - Building
Obtain an API key
It could be a lot better to have screenshots navigating to API key generation.

The sub-section inside the Building section is not needed, in my opinion.

With Google drive
With Discord, etc
These section shouldn't exist there, instead the content of these section could be put inside it's individual Providers section.

The Building section only consist the this page.
and this section should redirect to
each Providers respectively.
Means click on Google Drive With Google Docs should redirect to this page.
and the content should also move to this provider's page.

Starter Section:
the Get Started section should also have links to Step 1 - Setup and Step 2 - Building page for new project and api key guide.
for Understanding Semantic Search section I would rather prefer a common generic example for semantic search.
The Code for Understanding semantic search and Generative Search looks little complex and not explained well enough. Feels like there are lots of things going on and lots of things to process for the mind.

API --> Authorization
is that  ***CURL*** request working for you?
 API --> Search ---> Semantic Search
It would be better to have a good concise difference between **Lexical vs Semantic searching** in the first paragraph.
for this query:
unbody.get.<object_name>.search.about("query", {...options})
would be better to have a link pointing to what are available <object_names> and what are ...options

Docs search not working !

 API --> Search ---> Hybrid Search
I don't really get what exactly is Hybrid Search and what is it's exact use case, could be better to have a real example.
similarly examples should have exec function
The docs say about relativeScoreFusion argument for hybrid search but not sure what it is, and the SDK also doesn't have that property.

Tutorial:
The tutorial section better has something like:
How to build a simple TODO app with Unbody? How to build "Something" with Unbody?
It would be better to have a video form tutorial too.

This needs to be discussed further! while I agree with the idea of TODO but that is difficult to come up with such a example without making it too complex! how about we go around functionalities or features. for example "how to build  a semantic search" ? if not then we need to come up with super simple ideas; for example "build a move recommended system" ? and then it can evolve over multiple steps.

Docs search not working !

@jeangovil

API --> Search ---> Hybrid Search
I don't really get what exactly is Hybrid Search and what is it's exact use case, could be better to have a real example.
similarly examples should have exec function
The docs say about relativeScoreFusion argument for hybrid search but not sure what it is, and the SDK also doesn't have that property.

We have 3 search methods; semantic wich is base on concept and meaning, bm25 which is key word matching and then hybrid which combines these two methods.


Unable to view the assets of the Sc1 map.(Scx)

cantview.zip
This map is protected through a tool called Freeze.
https://github.com/armoha/euddraft (Here is a file called freezeMpq)
When I open the map with mpq editor, I can't see the asset completely.
What should I do in this case?

You will need a listfile for that. I tried mine and I can at least see staredit\scenario.chk.

https://github.com/ladislav-zezula/StormLib/issues/231
I saw the article here and used the scanner, but the computer was frozen with a memory error.
My guess is because maps made with the latest EUD editor are not compatible with the 1.16.1 version.
Even with eudenabler.dll, only the EUD maps created before the remaster are compatible, and the maps created after the remaster are incompatible.
Please, I hope the scanner in the MPQ editor can be made compatible with the scremaster.

Old EUD map

EUD Map Created with Latest Editors

I'll check the name scanner with the map you provided in the first post.

I tried the map that you posted in the name scanner. It ended up normally. I need to say that name scanner is only designed to work with Warcraft III maps.
Which map failed for you?
Also, what is EUD?

After some consideration, I decided not to implement Starcraft Name Scanner. It is not worth the time.


Site doesn't build on latest update, another issue

Describe the bug
I used hugo mod get -u for blowfish to get the latest version as I'd seen there had been an update. After that I tried to deploy my page but ran into an error for building the page.
Error Code (Shortened to final error line as it's repetitive):
Error: Error building site: failed to render pages: render of "page" failed: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.30.0/layouts/_default/baseof.html:9:4": execute of template failed: template: _default/single.html:9:4: executing "_default/single.html" at <partial "head.html" .>: error calling partial: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.30.0/layouts/partials/head.html:48:48": execute of template failed: template: partials/head.html:48:48: executing "partials/head.html" at <resources.Concat>: error calling Concat: slice []interface {} not supported in concat

Expected behavior
Page build
Desktop
Error has issue from github actions for build and deploy, unsure of OS/Browser option here
Hugo & Blowfish versions
Hugo: 1.111.2/1.111.3 Extended
Blowfish: 2.31.1
More Context
I was 2 commits behind the latest before updating as I'd only started using it the other day, but now it's stopped building
The action for context:
# Sample workflow for building and deploying a Hugo site to GitHub Pages
name: Deploy Hugo site to Pages

on:
  # Runs on pushes targeting the default branch
  push:
    branches:
      - main

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
permissions:
  contents: read
  pages: write
  id-token: write

# Allow one concurrent deployment
concurrency:
  group: "pages"
  cancel-in-progress: true

# Default to bash
defaults:
  run:
    shell: bash

jobs:
  # Build job
  build:
    runs-on: ubuntu-latest
    env:
      HUGO_VERSION: 0.111.2
    steps:
      - name: Install Hugo CLI
        run: |
          wget -O ${{ runner.temp }}/hugo.deb https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_linux-amd64.deb \
          && sudo dpkg -i ${{ runner.temp }}/hugo.deb
      - name: Install Dart Sass Embedded
        run: sudo snap install dart-sass-embedded
      - name: Checkout
        uses: actions/checkout@v3
        with:
          submodules: recursive
          fetch-depth: 0
      - name: Setup Pages
        id: pages
        uses: actions/configure-pages@v3
      - name: Install Node.js dependencies
        run: "[[ -f package-lock.json || -f npm-shrinkwrap.json ]] && npm ci || true"
      - name: Build with Hugo
        env:
          # For maximum backward compatibility with Hugo modules
          HUGO_ENVIRONMENT: production
          HUGO_ENV: production
        run: |
          hugo \
            --gc \
            --minify \
            --baseURL "${{ steps.pages.outputs.base_url }}/"
      - name: Upload artifact
        uses: actions/upload-pages-artifact@v1
        with:
          path: ./public

  # Deployment job
  deploy:
    environment:
      name: github-pages
      url: ${{ steps.deployment.outputs.page_url }}
    runs-on: ubuntu-latest
    needs: build
    steps:
      - name: Deploy to GitHub Pages
        id: deployment
        uses: actions/deploy-pages@v1

Full Error
hugo v0.111.3-5d4eb5154e1fed125ca8e9b5a0315c40dab2+extended linux/amd64 BuildDate=23-03-12T11:40:50Z VendorInfo=gohugoio
ERROR 2023/03/14 02:55:14 render of "page" failed: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/_default/baseof.html:9:4": execute of template failed: template: _default/single.html:9:4: executing "_default/single.html" at <partial "head.html" .>: error calling partial: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/partials/head.html:48:48": execute of template failed: template: partials/head.html:48:48: executing "partials/head.html" at <resources.Concat>: error calling Concat: slice []interface {} not supported in concat
ERROR 2023/03/14 02:55:14 render of "page" failed: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/_default/baseof.html:9:4": execute of template failed: template: _default/single.html:9:4: executing "_default/single.html" at <partial "head.html" .>: error calling partial: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/partials/head.html:48:48": execute of template failed: template: partials/head.html:48:48: executing "partials/head.html" at <resources.Concat>: error calling Concat: slice []interface {} not supported in concat
ERROR 2023/03/14 02:55:14 render of "page" failed: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/_default/baseof.html:9:4": execute of template failed: template: _default/single.html:9:4: executing "_default/single.html" at <partial "head.html" .>: error calling partial: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/partials/head.html:48:48": execute of template failed: template: partials/head.html:48:48: executing "partials/head.html" at <resources.Concat>: error calling Concat: slice []interface {} not supported in concat
ERROR 2023/03/14 02:55:14 render of "page" failed: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/_default/baseof.html:9:4": execute of template failed: template: _default/single.html:9:4: executing "_default/single.html" at <partial "head.html" .>: error calling partial: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/partials/head.html:48:48": execute of template failed: template: partials/head.html:48:48: executing "partials/head.html" at <resources.Concat>: error calling Concat: slice []interface {} not supported in concat
Error: Error building site: failed to render pages: render of "page" failed: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/_default/baseof.html:9:4": execute of template failed: template: _default/single.html:9:4: executing "_default/single.html" at <partial "head.html" .>: error calling partial: "/tmp/hugo_cache/modules/filecache/modules/pkg/mod/github.com/nunocoracao/blowfish/v2@v2.31.1/layouts/partials/head.html:48:48": execute of template failed: template: partials/head.html:48:48: executing "partials/head.html" at <resources.Concat>: error calling Concat: slice []interface {} not supported in concat
Total in 19562 ms
Error: Process completed with exit code 255.

Hey @joush007 this doesn't sound like a bug. Check the workflow files in this repo for Blowfish's site. I am also using GitHub actions and it's working. There must be some issue in your flow. Happy to help but in the discussion forum not in a ticket, especially one with a dramatic title like this one.

I would check any overrides you created in your site

Alrighty gotcha, sorry about that, but also what do you mean by the overrides?

Alrighty gotcha, sorry about that, if I have any other questions about it I'll move it to a discussion forum, but I'm wondering what you mean by the overrides I've created in the site?

Overrides might change how files get referenced, or dependencies. Quick test would be to remove overrides and to one test. Tag me on the discussion if that doesn't work. Happy to help


Fixed Checkstyle issues:

1.Line has leading space characters; indentation should be performed with tabs only.
2. Line does not match expected header line of '^\Q *
3. Wrong order for 'java.util.ArrayList' import. [ImportOrder] 'XYZ' (e.g. 'org.apache.commons.logging.Log') should be separated from previous imports.
4. Block tags have to appear in the order '[@param, @author, @SInCE, @see, @Version, @serial, @deprecated]'. [AtclauseOrder]
5. '}' at column 17 should be alone on a line. [RightCurly]

Checked and it looks good!
Build without test does not fail anymore due too community styles check!
./gradlew spring-integration-nats:build -i -x test
BUILD SUCCESSFUL in 5s
22 actionable tasks: 3 executed, 19 up-to-date
Watched directory hierarchies: [/~/Work/github/rohlenko/spring-integration]
That is why pull RQ going to be merged.


Handle overlapping ranges

This assert triggers for some binaries.
May also need better validation of ranges in general. May also want to ignore ranges that begin at 0. Also see #49  for some changes that may be required.

Hm. I did run across some invalid ranges like the 1..1 you mention in #49, which is why I'm trying to filter those out with the start != end check. Perhaps there is a way to identify the other invalid ranges as well?
If there really is no other way, we could use an interval tree here as well, but I'd really like to avoid dropping unit ranges entirely. Is there any legitimate reason for multiple compilation reasons to describe the same address ranges?

My memory of #49 is that we failed to return a result for some addresses. The line table had entries for the addresses, but the unit ranges were missing. I'll try to reproduce this again.
I don't think there's any legitimate reason for multiple CUs to cover the same range. I've seen many invalid ranges starting at 0 though, which should be easy to filter out if needed.
In theory, the unit ranges and the line ranges should be identical, so the only advantage of the unit ranges is they are faster to parse, but if they do match the address then we need to parse the line ranges anyway. Also, it's the line ranges that have the information we want, so they should always be more accurate. There's also .debug_aranges which should be an even faster map of address to unit, but I think it's often broken (this is what eu-addr2line uses, and I think that's why it doesn't work).

AFAIK .debug_aranges is optional and many compilers just don't generate it at all (I think clang does but rustc doesn't).
Unit ranges allow lazy parsing of units (line number program), I think that may be a nice optimization.

I'm seeing this on nightly build (https://circleci.com/gh/sfackler/rstack/23), but not a stable build of the same commit (https://circleci.com/gh/sfackler/rstack/22). Might be caused by https://github.com/rust-lang/rust/issues/45511?

Using a nightly older than nightly-2017-10-24 didn't fix the problem for rstack, so it's not https://github.com/rust-lang/rust/issues/45511.

For rstack, the issue is a couple of compilation units that use low_pc/high_pc to specify the range, but the low_pc is 0. Both units contain a single subprogram that also has a low_pc of 0. I suspect this is similar to the 1..1 ranges that occur elsewhere, but it gets handled wrongly because there's only one subprogram in the unit.
For the other binaries I've tested on, there are subprograms that have debuginfo in multiple units, and this is a relatively common occurrence. These seem to all be C++ constructors/destructors, so I suspect this is the compiler generating multiple copies and combining them when linking.

Investigating #49 again, I was mistaken about what was happening there. Here's the problem cases I have found.

A unit may specify DW_AT_low_pc = 0, and have no DW_AT_high_pc or DW_AT_ranges, but still have valid line sequences. I've seen this with clang 3.4. I don't care too much about this one, but other tools handle it, and if we can get it for free then we should.

A unit may contain a single subprogram, and both the unit and subprogram specify DW_AT_low_pc = 0, and .debug_lines has a sequence starting at 0. This was seen for rstack.

A function may have DW_AT_low_pc = 0, and a corresponding line sequence starting at 0. Inlined subroutines within that function are relative to that, so they are also invalid addresses, but they may not start at 0. This is a more general case of (2), and doesn't trigger the assert because it's not in the unit ranges.

A function may appear in multiple units. Usually the address range for the function will be identical in every unit, but not always, and I don't know how to determine which one is valid in that case. I think this occurs for automatically generated code in C++ (eg destructors).

For rust, if LTO inlines into main or __rust_oom then there are line sequences for the inlined functions, but nothing in .debug_info. This could probably be fixed in rust by specifying debug metadata for main and __rust_oom, but I'm not sure if this can happen in more instances.

libxul.so from firefox 56.0+build6-0ubuntu<IP_ADDRESS> has some weirdness in one unit generated by rustc 1.17.0. Some functions are missing from .debug_info, and the line sequences only cover the tail end of the functions. Not a problem for firefox 57.0, so probably can ignore this one.

So I think 3 changes are possible:

remove the assert, to allow for (4). This happens for functions that are actually the same, so I don't think this affects correctness.
stop using unit ranges, and use only line sequences, to allow for (1) and (5). Neither are high priority to fix, but I don't see any downside to doing this.
ignore line sequences and functions with address 0, to allow for (2) and (3). Ignoring inlined subroutines in this case may be tricky. This change isn't really necessary if you only ever look up valid addresses, but with the currently algorithm, looking up an address such as 0 or 1 can give you a very long call trace, since it thinks there are many functions at that address.

Unit ranges allow lazy parsing of units (don't look at the line number program until you need it), I think that may be a nice optimization.

This is true for the first lookup of an address that isn't in that unit. But lookups of addresses in that unit will be slower both for the first and subsequent lookups since you need to do a binary search for both the unit ranges and the line sequences (unless you discard the unit ranges once you have parsed sequences).


Add Gem ranking badges

This pull request adds badges for Gems ranking, taken from bestgems.org:

/gem/rt/<gem> for total ranking (consolidated)
/gem/rd/<gem> for daily ranking

After the facts, I think I would consider total ranking the default, and avoid adding total in the badge. Do you agree? Can I remove it?

Sounds reasonable
On Wed, 13 May 2015 08:46 Thaddee Tyl<EMAIL_ADDRESS>wrote:

After the facts, I think I would consider total ranking the default, and
avoid adding total in the badge. Do you agree? Can I remove it?
—
Reply to this email directly or view it on GitHub
https://github.com/badges/shields/pull/433#issuecomment-101536833.


Cannot read property 'type' of undefined

Hi,
My date time picker display error when i close the picker after select date and time
i enclosed that screenshot please any one find out the solution for that.

@sureshhustle thanks for reporting this, but could you add a plunker example to reproduce this?

I am also facing same problem, its coming in your example plunker, it came only first time,
https://plnkr.co/edit/J6hXyB?p=preview

Fixed


git extract: remove dialog in favor of text editor

I'd suggest moving to an editor where an editor is opened with the list of commits and the user deletes all but the commits they want to extract. This is similar to the git rebase interactive workflow. It removes a dependency and makes the user interface closer to the other git commands

+1 I agree

Sounds good to me

This ticket is obsolete in the light of #601. Ok to close?

Closing as git-extract has been removed


depthai builds failing for armv6l (raspberry pi zero). Last successful build was 2.13

Raspberry pi zero builds of depthai (armv61) last successfully built with 2.13 (about 7-8 months ago). Will it not be supported for future builds or is there some other reason for this? This surfaced for me as I'm trying to use a oak-d-pro (the IR light requires depthai 2.15 or above) with a rpi zero for testing

Great question.  Will have to check with the team.  I'm not immediately sure what we use as the runner for the Pi Zero.  Will circle back.

@themarpe Thank you! I just checked for the new release (https://www.piwheels.org/project/depthai/) and looks like it still didn't build.

@bgro82 sorry for the issue - looks like we'll have to add our own runners to cover this case...

@themarpe thanks, do you know when this will be fixed?

@bgro82 unfortunatelly piwheels builds still have issues with latest release.
We'll likely have to run this on our own runners - currently we only build for ARMv7. Preferably we'd build for ARMv6 on a stronger RPi4B, but we've yet to implement this.
Any pointers are appreciated!

@themarpe any ideas on what may be still causing the builds to fail? I primarily bought these cameras to use with the rpi zero but can't use a recent version of depthai (which causes other dependency issues, etc)

after some more reading, is it possible that 32 bit support was removed or disabled in some way?
also after more reading, it appears the architecture is armv6l (not armv61).

@bgro82
Yeah, the armv6 builds aren't available at the moment.
The issue is also that building natively on the Pi Zero isn't easy due to limited amounts of RAM.
I'll see about spinning up a crossbuild this weekend.

great thank you. Is it possible to use more disk swapping to make up for the low RAM (understanding it would be much slower)?

@bgro82 yes it would be possible.
In my limited testing i did, i haven't been able to setup a proper cross compilation setup for it, but if you opt for swap, you should be able to compile it as intended

I increased my swap file to 1024M (default on the pi zero is 100M) and was able to build this successfully when including "--no-dependencies"

If there is somewhere I can share the wheel file let me know and happy to post it to save others the build time.

That'd actually be great - did you compile 2.20.2? Feel free to share here / link to it.

sure thing. Here is the latest depthai and opencv built for armv6l for raspbian bullseye and python 3.9:
depthai-<IP_ADDRESS>-cp39-cp39-linux_armv6l.whl: https://drive.google.com/file/d/1HWVVFAy7eKKuxEpd1KGq2tl71bJR4kKQ/view?usp=sharing
opencv_python-<IP_ADDRESS>-cp39-cp39-linux_armv6l.whl: https://drive.google.com/file/d/1SONV5cnrZV05fSqLexs-MDXxZyJch0x_/view?usp=sharing

Thanks @bgro82  - checked and available over at: https://artifacts.luxonis.com/artifactory/luxonis-python-release-local/
The below command should work as expected
python3 -m pip install --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-release-local/ depthai

Excellent, thank you. Happy to build these and provide from time to time as well if they are useful to others.

Hi @bgro82 @themarpe,
it would be great if you could build and provide armv6l wheels for the latest depthai release (v<IP_ADDRESS>). This would enable users of Raspberry Pi Zero W (v1) to use the latest features.
Thank you!

@maxsitt
Do you mind doing the following:

Download https://artifacts.luxonis.com/artifactory/luxonis-python-release-local/depthai/depthai-<IP_ADDRESS>-cp39-cp39-linux_armv7l.whl (bullseye) or https://artifacts.luxonis.com/artifactory/luxonis-python-release-local/depthai/depthai-<IP_ADDRESS>-cp37-cp37m-linux_armv7l.whl (buster)
rename from linux_armv7l to linux_armv6l
python3 -m pip install [wheel]

If that works, I'll add this as part of our releases (cross checked, our runners build with march=armv6 already, so I think this should be ifne)

After you added the (renamed) armv6l wheel it will be possible to install the depthai package on RPi Zero W with a simple python3 -m pip install depthai? Or do we still have to add --extra-index-url https://artifacts.luxonis.com/artifactory/luxonis-python-release-local/ ?

According to some pypi discussion, it doesn't allow uploading both armv6l and armv7l wheels, but I'll give that a go.
Otherwise, it'll be available under our releases. We'll update the docs accordingly.
And thanks for giving this a go!

Thanks a lot @themarpe !
When running python3 -m pip install depthai==<IP_ADDRESS> on RPi Zero W, depthai-<IP_ADDRESS>-cp39-cp39-linux_armv7l.linux_armv6l.whl is downloaded and successfully installed. Everything I tested works perfectly (custom scripts with YOLOv5n model detection & tracking).

Closing as addressed now:)


update ubuntu 18.04 to ubuntu 20.04

the base image version is inconsistent between Dockerfile and README
update all ubuntu 18.04 to ubuntu 20.04

/test-e2e

/skip-all

@shadowlan the author information in your commit is missing. This means that if I merge this as is, you may not be given credit on Github for the contribution. You may want to add the email address you used for the commit to your Github account. As far as I can tell, the address is<EMAIL_ADDRESS>

@antoninbas , thanks, I will check and use my personal email.

@antoninbas  considering it will require force push for a new commit message,I just add my work email to my account, you can merge it now, thanks!


Discuss upstream contributions from CD with package managers

@pombredanne mentioned it would be a good idea to reach out to the package managers about how to respectfully and tastefully start to work with upstream projects for the ClearlyDefined data once it starts to get curated.
We need the project to move a bit further along and to have more substantial harvested and curated components, but this is something we should keep an eye on and check in about soon.

Given that this outreach is something that we'll have to constantly address, I'm closing this issue for now.


[Synapse] - General Available Release

Code generated based on the Swagger spec: https://github.com/Azure/azure-rest-api-specs/blob/master/specification/synapse/resource-manager/readme.md
Tag: package-2020-12-01

@idear1203 Could you link the swagger PR that your updates are associated with?

@idear1203 Could you link the swagger PR that your updates are associated with?

Related PR: https://github.com/Azure/azure-rest-api-specs/pull/12502

@idear1203 Please:

update versions in *.csproj and AssemblyInfo.cs, and follows semver
update release notes in *.csproj
update API tags in Az.Sdk.props if there's a API-version update
make sure generated code is not touched
make sure has proper test coverage

@idear1203 Please fix the hanging check-enforcer.


latest power toys preview won' dispay settings dialogue

No access to settings. displays an unpopulated settings window for mabe a fifth of a second and teh its gone. Power Toys continues to work other than no access to settings. This issue survived an OS reboot.
PowerToysReport_2021-10-31-13-25-16.zip

Could you attach a full bug report file?
Right-click on the tray icon and click Report Bug. Then drag-and-drop the generated file into the GitHub comment to upload it. Thanks!

User already sent it.
@Bluto39  It seems you're still on 0.45.
Would you mind:
1 - Uninstall PowerToys.
2 - See if the PowerToys install directory (default: C:\Program Files\PowerToys) is still there. If it is, delete it manually.
3 - Install PowerToys.
Please let us know if this works for you.


Cant find protoc?

~/tiles-chain/dappchain]$ make protoc --plugin=./protoc-gen-gogo -Isrc -I/usr/local/include --gogo_out=src src/types/types.proto make: protoc: No such file or directory make: *** [src/types/types.pb.go] Error 1

You need to have Protobuf binary from Google installed, https://loomx.io/developers/docs/en/prereqs.html


Question about how to skip foreign layer?

Question about how to skip foreign layer or how to speed up docker push?
Hi,
I see the announcement provided Announcing Removal of Foreign Layers from Windows Container Images in previous issue, which said about the foreign layer would be added. Actually as my own experience, this change is very unfriendly because  it made docker push very slow. In our case, the duration of docker push in worst case is ten times than before, and usually about five times than before. So my question is there any way to skip push the foreign layer? For example, does there exist command to skip push foreign layer during the docker push? Or do you have other idea and solutions?
Thanks for your time,
Chenxi

Hi team, any suggestions about it?

Hi, maybe does anyone has any idea?

Question about how to skip foreign layer or how to speed up docker push? Hi, I see the announcement provided Announcing Removal of Foreign Layers from Windows Container Images in previous issue, which said about the foreign layer would be added. Actually as my own experience, this change is very unfriendly because it made docker push very slow. In our case, the duration of docker push in worst case is ten times than before, and usually about five times than before. So my question is there any way to skip push the foreign layer? For example, does there exist command to skip push foreign layer during the docker push? Or do you have other idea and solutions? Thanks for your time, Chenxi

Hey, I have a few questions and items I want to clarify first. I just want to clarify that the foreign layer has been removed and no longer exists.
What is the size that you are pushing?
What is your internet speed?

For your question, the size is 3.6GB, and because we are using azure resources that is the VM created by image with D8s sku.
The internet speed is not stable, sometime push all layers to azure cr takes 4mins, but the worst case takes 30mins, and for now the P80 of the pushing time is 8mins, but our build pipeline just 15mins without push image.

Hi @NAWhitehead ,For your question, the size is 3.6GB, and because we are using azure resources that is the VM created by image with D8s sku. The internet speed is not stable, sometime push all layers to azure cr takes 4mins, but the worst case takes 30mins, and for now the P80 of the pushing time is 8mins, but our build pipeline just 15mins without push image.

Hello, when we decided to remove the foreign layer from the container images we knew that there would be an increased start-up cost for those that had unstable internet connect for the first image push. However, now that you have pushed to your container registry the next time the image will be cached in your repository and only the new added layers will need to be pushed going forward.

Hi @NAWhitehead ,For your question, the size is 3.6GB, and because we are using azure resources that is the VM created by image with D8s sku. The internet speed is not stable, sometime push all layers to azure cr takes 4mins, but the worst case takes 30mins, and for now the P80 of the pushing time is 8mins, but our build pipeline just 15mins without push image.

Hello, when we decided to remove the foreign layer from the container images we knew that there would be an increased start-up cost for those that had unstable internet connect for the first image push. However, now that you have pushed to your container registry the next time the image will be cached in your repository and only the new added layers will need to be pushed going forward.

But now, we're facing a problem that is the push container image step is always push all layers, even if there already has same layer. Hi @NAWhitehead do you know what will cause it?

Hey, we are going to try and replicate the issue and I will get back to you.

But now, we're facing a problem that is the push container image step is always push all layers, even if there already has same layer. Hi @NAWhitehead do you know what will cause it?

Hey we are going to try and replicate this issue and I will get back to you.

But now, we're facing a problem that is the push container image step is always push all layers, even if there already has same layer. Hi @NAWhitehead do you know what will cause it?

Hello, we have tried recreating this could you provide your steps so that we can follow what you are doing so that we can attempt to recreate this issue.

Closing issue because it's going stale.


How to use keyword search and exact phrase match

For example, I want to search for "lionking simba", what is the code for using keyword? What is the code for using exact phrase matching? And I only want to search for original tweets, how do I add parameters? Could you please show an example

Hello,
Your question doesn't have anything to see with this library. You should check the Twitter API documentation here instead


Starting 2019 CANADA BC Family Day will be 3rd Monday of Feb

Thank you very much for your contribution.
Please fix Travis CI errors in order to have your PR merged.
Thx
dr-p

Hi dr-prodigy,
I don't quite understand the errors on my second commit.
"File "tests.py", line 440, in test_family_day
self.assertNotIn(dt, bc_holidays)
AssertionError: datetime.date(2019, 2, 18) unexpectedly found in {datetime.date(2006, 7, 1): 'Canada Day', datetime.date(2000, 12, 26):......
Any ideas?
Thanks.

Nvm, the test.py needs to be updated. New to GitHub, will try to see if how do I do that.

Ok, I think this build will work. Sorry that I bombard your CI with minor changes; I don't have the IDE setup where I am now.
Thanks,
Joe

Pull Request Test Coverage Report for Build 329

12 of 12 (100.0%) changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.002%) to 98.947%

Totals

Change from base Build 321:
0.002%

Covered Lines:
4604

Relevant Lines:
4653

💛  - Coveralls

Thank you! 👍


Does GCS fetching result in a lot of extra bandwidth?

I'm reading through the code to try and calculate how many remote store calls one would expect from a normal Kafka deployment.
As far as I trace the read path, I notice that for each chunk we call a GET separately. This get is for the particular chunk - i.e the byte range. While the S3 and Azure implementations use the particular API's range fetching capabilities - the GCS one doesn't (at least not obviously).
Reading further into it, I see that the way to do range fetches in the Java API is through the storage.reader() API, and I notice that Blob.reader() does in fact just do that.
So I guess this answer my question with a No. We only fetch the bytes of the range we request.
Additionally, if I may ask:

Is it a correct understanding that we do a GET for each chunk?

If yes, have we considered GET-ting the whole object (or at least a larger part) and then caching the chunks (at least when a cache is configured?). Since consumers/brokers typically read linearly when reading historically, you'd expect that if you fetch a 4MiB chunk of a segment, you're probably gonna fetch the full log segment anyway

We PUT the whole segment log in multipart chunks via s3.multipart.upload.part.size / gcs.resumable.upload.chunk.size / azure.upload.block.size but GET it in chunk.size.

Is there a reason these weren't made to be more consistent with one another? At least with the default?

A third question, perhaps for @ivanyu, the code says:
 private Blob getBlob(final String key) throws KeyNotFoundException {
        // Unfortunately, it seems Google will do two a separate (HEAD-like) call to get blob metadata.
        // Since the blobs are immutable in tiered storage, we can consider caching them locally
        // to avoid the extra round trip.
        final Blob blob = storage.get(this.bucketName, key);

But EXTENSIVELY searching through Google, reading through the documentation, reading the GCS Java client code - I don't find any reference of this. Is it still true?

Fourth question: The default size for a segment is 1 GiB, so the way this behaves seems to be:

Kafka awaits for the active segment to get rolled (1GiB) and LSO to pass.
Kafka starts uploading to S3 via this plugin - 1 multipart size at a time (5 MiB), serially in the same thread.
1 GiB / 5 MiB is 205 requests. If each have 200ms RTT, that's 41 seconds to fully upload. (per partition)

Is there any reason the upload size isn't larger by default?
AFAICT a big reason for multipart upload is to make uploads parallelizable (and retriable) - but since we don't leverage that here, it makes sense to chunk larger portions?

@jlprat @giuseppelillo thanks for merging my docs PR. Is it possible to get some attention on this? Just looking to have a discussion, not necessarily an outcome

Is there any reason the upload size isn't larger by default?
AFAICT a big reason for multipart upload is to make uploads parallelizable (and retriable) - but since we don't leverage that here, it makes sense to chunk larger portions?

Agree. We were considering to make these uploads parallel at some point, but decided against it as it would have use more IO/CPU. Instead we introduced a rate limiter to make the upload predictable.
About the size, we defaulted to the minimum to start with but I do agree with your analysis, and a higher default should be proposed. If you have a better value in mind, could you propose a PR? Would be much appreciated.

Correct. This is part of the pre-fetching functionality on the chunk cache

Yeah, what I meant by the question was whether that object get is many different get() calls, or one big one. I was suggesting we just call one GET to get all the data (saving remote calls)
--
Makes sense on the get size!
--

About the size, we defaulted to the minimum to start with but I do agree with your analysis, and a higher default should be proposed. If you have a better value in mind, could you propose a PR? Would be much appreciated.

Just looking at the range, it seems like it's standard practice to have larger chunk sizes.
This isn't a high priority issue, but increasing size 10x or so would cut down on Kafka's PUT request cost 10x too.
It's not too extreme - the costs are small nevertheless. But still - seems like an easy gain.
As an example, a 50 MB/s AWS cluster would average 9.6 PUT/s under this implementation and cost $1,520/yr for those puts. Changing it would reduce the cost to $152/yr. That being said, the 1.5k cost is just 1% of the total cluster cost, so you'd at best save 0.9% of the total Kafka cost - not a huge amount. Still seems like an easy improvement!

Thanks @stanislavkozlovski ! I have added a note to our docs to suggest tuning this config: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/pull/626