Additional parameters in some functions - new functions

Dear all,
I am writing to ask you some other functionalities.
I have just moved from Seurat to Scanpy and I am finding Scanpy a very nice and well done Python package.

I wrote a function to show the 3D plot of the UMAP, tSNE and PCA spaces. In the scanpy.tl.tsne function is not possible to change the number of components, it calculates only the first two components, even if the scanpy.pl.tsne function has a parameter component. May you add a parameter like the n_components of the scanpy.tl.umap function?

In the rank_genes_groups function the log2FC values are provided only for ‘t-test’ based methods. May you return the log2FC values (maybe named log2FC) for all the implemented statistical methods?

I think that two parameters in the rank_genes_groups function should be added.

min_pCells to test only the genes that are detected in a minimum fraction of cells of either of the two populations (e.g., cluster 0 vs rest). For instance, min_pCells=0.3 means that at least 30% of the cells must express that gene.
positive, if it is True, the function should return only positive marker genes for each population.

A function showing the volcano plots (based on the log2FC) can help (I can write it if the log2FC values are provided).

Thank you in advance.
Best,
Andrea

Thank you for the constructive wish list! All of it sounds doable. Would you feel comfortable with adding this functionality yourself? A pull request would be welcome!

Thank you for the prompt reply,  I am going to open a pull request.
I can work on points 1 and 4. May you help me with the other 2 points?
Thanks!

@andrea-tango
Really awesome!
I am also wondering to find some parameters to tune in scanpy's rank_genes_groups like in Seurat. Because I found there is some difference in makers by scanpy's default and Seurat's default parameters only.pos = TRUE, min.pct = 0.25, logfc.threshold = 0.25.
I can find interesting markers calculated by Seurat but not in Scanpy's.
Have you ever came into similar results?

Interesting, @MichaelPeibo! Would you share these results somewhere publicly? A notebook on GitHub?
@andrea-tango. Yes, we should get the functionality of 2 and 3 functionality into rank_genes_groups. @Koncopd, could you work on this?

@MichaelPeibo @falexwolf I started working on points 2 and 3, but it is better if you will work on these points.
I wrote the code for points 1 and 4.
In order to generate volcano plots, I calculated the log2FC relying on the diffxpy library.
I can push again the code for tSNE and also the code for volcano plots.
Please, check the rank_genes_groups function.
Considering 2 groups of cells and using the Wilcoxon test (de.test.wilcoxon) provided by the diffxpy library, I obtained different marker genes with respect to those calculated by using rank_genes_groups function (Wilcoxon test).
Many thanks.

@falexwolf
Hi, yes.

Great, thank you, @andrea-tango and @Koncopd!
@andrea-tango, would you make a PR? We can then look at how you solved this. In principle, I'm very hesitant to add diffxpy as a dependency of Scanpy. It depends on Tensorflow itself, which is a large dependency. What would be OK would be to have a wrapper in scanpy.external, but I don't know whether this makes sense. Why not using diffxpys Volcano plots right away?
Regarding the discrepancy between wilxocon in diffxpy and scanpy. There obviously shouldn't be any and there also shouldn't be duplicated code, here, at all. The only reason that Scanpy has its own Wilcoxon implementation was that there was no implementation available that would scale to large sparse data. That's why @tcallies wrote the present implementation about 1.5 years ago. He benchmarked with scipy's Wilcoxon test. @davidsebfischer, can you shed light on why and how you implemented your Wilcoxon? Shouldn't we have a comparison? At the time, @tcallies wrote this and these tests. How did you write your tests?
We were just talking about log2FC, which is such a simple quantity and should evidently be properly computed by rank_genes_groups. We just had this other PR on it  (https://github.com/theislab/scanpy/pull/519).
@tcallies, any thoughts from your side?

@falexwolf I agree with you about the diffxpy  a scanpy dependencies, Tensorflow is a very important dependency!

would you make a PR?
I did it, I pushed the code where I added the parameter n_components for scanpy.tl.tsne function.

Why not using diffxpy Volcano plots right away?
I wrote a function in which you can change the colour of the genes, you can add the names of the genes etc.

How did you write your tests?
I tried them on data coming from the lab in which I am working.
I can write a jupyter notebook using public dataset and push it on my copy of the scanpy repository.
Give me a couple of days.

Oh, thanks! Sorry for the long downtime, the whole family was sick... I'm going through the PR now.
The tests question was actually targeted towards @davidsebfischer, but thanks anyways! The comparison question was also targeted to @davidsebfischer, @tcallies. But if you do it, @andrea-tango, awesome!

Very good suggestions. One small question:

positive, if it is True, the function should return only positive marker genes for each population.

Isn't this rankby_abs=False which we have by default in rank_genes_groups? (and which was actually called only_positive=True previously https://github.com/theislab/scanpy/blob/master/scanpy/tools/_rank_genes_groups.py#L100)

Hi @falexwolf
I am sending you the simpler code ,my anndata which can be reproduced in some way, and csv marker file calculated by Seurat to your email, you might to repeat my marker analysis if you would like
Sorry for doing it in this way, I not familiar about how to make public notebook...and our data is too preliminary to be public.

Hi all, thanks for the mention @andrea-tango! If have been making multiple changes in diffxpy and batchglm recently, the following refers to the branch diffxpy dev, I haven' merged all of this into master yet as I am waiting for some last issues to be fixed.

Are diffxpy's tests correct: We are using unit tests to check a) that all tests fullfill the null model. b)  for standard tests that do not require model fits (Welch's t-test and rank sum test) we check that we produce the same results as scipy. We vectorise where possible but we do actually directly call scipy in the rank sum test right now. Both of these tests check out on the dev branch for the rank sum test, so this is working correctly. @falexwolf this might be a unit test that you could also introduce? I did not see this in the ones that you linked but I just glanced over. @andrea-tango please use dev right now.

Nature of the rank sum test. We previously called the Wilcoxon rank sum test de.test.wilcoxon, note that this is also alternatively named Mann-Whitney U test (MWU). Importantly, MWU is for independent samples, which we always have in scRNAseq, the "wilcoxon" test in scipy is for paired samples. We therefore renamed this to de.test.rank_sum to avoid confusion. Which one are you using @falexwolf?

Comparison scanpy vs diffxpy (in unit tests). I would discourage this and compare against scipy because I would consider scipy the gold standard for statistical testing. I can run comparisons if the above comments do not resolve all issues discussed here, which would imply that we might differ in our wrapping in the tests.

Hi @davidsebfischer, I am writing a simple jupyter notebook where I am analysing the 10x_pbmc68k_reduced.h5ad data.
I selected only clusters 0 and 1:
twoClusters = adata[np.logical_or(adata.obs.louvain == '0', adata.obs.louvain == '1')]
Running  sc.tl.rank_genes_groups(twoClusters, groupby='louvain, method='wilcoxon',  corr_method=''bonferroni), I obtained the following genes

Trying with diffxxy library,
test = de.test.wilcoxon(data=twoClusters, grouping="louvain")
there is the following error: All numbers are identical in mannwhitneyu

@andrea-tango please use dev right now.
For this test, I used the version downloaded with pip.
I can clone the repository and use the diffxpy dev branch.

Just a note... I also have some code to make a volcano plot in line [111] here. Don't know if this is still needed, but I thought I would see if someone cares.
@davidsebfischer do you allow labels in your volcano plot function? And can it take any object, or only some custom diffxpy ones? Then I'd prefer to use yours tbh.

@andrea-tango @MichaelPeibo To address the filtering of rank_genes_groups (eg. min.pct = 0.25, logfc.threshold = 0.25) I recently added a function called sc.tl.filter_rank_genes_groups.  See https://github.com/theislab/scanpy/pull/425
@falexwolf I don't know whysc.tl.filter_rank_genes_groups does not show up in the docs. I will take a look. Also, I just noticed that this PR with updated examples is still open. I think it would be useful to merge: https://github.com/theislab/scanpy_usage/pull/11

does sc.tl.filter_rank_genes_groups filter the sc.tl.rank_genes_groups result? Or does it recompute? The former would not alleviate the multiple testing burden.

does not recompute, simply saves the filtered data under
adata.uns['rank_genes_groups_filtered']. Thus, different parameters can be
tested quickly. Off course, sc.tl.rank_genes_groups has to be call first.
On Mon, Mar 11, 2019 at 3:47 PM MalteDLuecken<EMAIL_ADDRESS>wrote:

does sc.tl.filter_rank_genes_groups filter the sc.tl.rank_genes_groups
result? Or does it recompute? The former would not alleviate the multiple
testing burden.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/theislab/scanpy/issues/460#issuecomment-471569285,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEu_1debC8DotLkQywhO8zJpEvfkBbSHks5vVmxpgaJpZM4ahuSs
.

If p-values are regarded as a valuable output rather than just the ranks, it might be worth recomputing as thresholding would ease the multiple testing burden. I guess that's the idea behind the min_pCells parameter request.

@LuckyMD your tutorial is very interesting!
I agree with you regarding the min_pCells parameter, it should be used as a filtering step before calculating the marker genes.

@falexwolf @andrea-tango
I have a question regarding point 2 (log2FC values in rank_genes_groups). I see that only 'logreg' method doesn't return logfoldchanges. But logfoldchanges don't seem natural for 'logreg' as this method doesn't even use reference for calculation.

Sorry this is a little off topic, but it's something I've found useful:
@LuckyMD and anyone else looking for an interactive volcano plot, I've been using hvplot in my notebooks. A volcano plot can be made from DE data frame with something like:
de_df.hvplot.scatter(
    "logfoldchanges", "pvals_adj",
    flip_yaxis=True, logy=True,
    hover_cols=["names"]
)

 Complete example using scanpy
import pandas as pd
import numpy as np
import hvplot.pandas
import scanpy as sc

def rank_genes_groups_df(adata, group, pval_cutoff : float =None, logfc_cutoff=None):
    d = pd.DataFrame()
    for k in ['scores', 'names', 'logfoldchanges', 'pvals', 'pvals_adj']:
        d[k] = adata.uns["rank_genes_groups"][k][group]
    if pval_cutoff is not None:
        d = d[d["pvals_adj"] < pval_cutoff]
    if logfc_cutoff is not None:
        d = d[d["logfoldchanges"].abs() > logfc_cutoff]
    return d

pbmcs = sc.datasets.pbmc68k_reduced()
sc.tl.rank_genes_groups(pbmcs, "bulk_labels", n_genes=pbmcs.var_names.size)
de_df = rank_genes_groups_df(pbmcs, "CD34+")

de_df.hvplot.scatter(
    "logfoldchanges", "pvals_adj",
    flip_yaxis=True, logy=True,
    hover_cols=["names"]
)

@ivirshup, this is cool and very useful. Would you make a notebook "Interactive plotting" for http://scanpy-tutorials.readthedocs.io? And we'd link to it from https://scanpy.readthedocs.io/en/latest/tutorials.html as done for the other notebooks. You could also add the example from https://github.com/theislab/scanpy/issues/510#issuecomment-473506882. As time passes, this could grow. But these two little examples are a very useful start, I think.

@Koncopd, updating logreg with a proper implementation accounting for reference is another story. We can talk about it sometime soon.


Handle errors with enums instead of Box<dyn Error>

When handling errors via Box<dyn Error>, you get a ton of boilerplate. You need to copy paste new error structs, you need to implement Display for all of them, etc. Instead, you should just handle errors with enums, and for printing out the details just implement Display for the error enum in a single place. That's literally it, you don't need anything else.

Implementing errors with Enums instead of the Error trait actually turned out to be easier than expected. Plus, it cut the error handling code by half. I shudder to think how much junk boilerplate the error code would have been handling once there were ~100 error types or more. I'm glad I caught this at this early stage.


[QUESTION] What's the themesource that supports Xbox Gamebar theme?

I noticed that all Microsoft made widgets support the theme specified by a user in the gamebar settings. What's the themesource that developers should use to support the theme mentioned?

XboxGameBarWidget exposes an ElementTheme property and event for when it changes.
https://docs.microsoft.com/en-us/gaming/game-bar/api/xgb-widget#game-bar-state

Currently the solution seems to be to write your own stylesheet.

I remember struggling to try and implement theming. I got somewhere by setting RequestedTheme on each Page element, but I ran into some problems (which I can't recall).
I would appreciate a more realistic example of how we're supposed to implement theming, rather than hard coding individual colours in the code-behind event handler

I would recommend following the general guidance for UWP apps and how to implement theming. Please take a look at this documentation to see if it helps better understand how to do this:
https://docs.microsoft.com/en-us/windows/uwp/design/controls-and-patterns/xaml-theme-resources
I agree with you that the samples could do a better job of showing how to implement theming properly. I can get a work item filed to improve that in the future. Please let me know if you have any more questions.


loadstring has invalid signature in the std libs annotations

Environment

name
version

IDEA version
2020.3 (IC-203.5981.155)

Luanalysis version
1.2.1

OS
Linux Ubuntu 20.10

What are the steps to reproduce this issue?

Use local l, s = loadstring(str) (the important point being s as 2nd retval)

What happens?
Error on s local variable: Too many assignees, will be assigned nil.
What were you expecting to happen?
No such warning, the 2nd retval is actually valid in case of failure: returns nil plus the error message (see the load doc which loadstring refers to).

Thanks!

Thanks!


Rspamd tries to DKIM sign incoming emails

Classification

[X] Question
[?] Minor bug

Reproducibility

[?] Sometimes

Docker information
docker info
Server Version: 17.05.0-ce
Kernel Version: 4.4.0-97-generic
Operating System: Ubuntu 16.04.4 LTS

docker images hardware/mailserver --digests --filter "dangling=false"
hardware/mailserver   1.1-latest          sha256:f3a51071cf0fac8334a119037f18979                                                                                b7d5f9d1ad968ed042ea668a57400f01c   de2385334fb9        7 days ago          334M                                                                                B

Description
Rspamd tries to DKIM sign incoming emails, fails to find key (expected), mails still get delivered.
Happens only with domains that have a DKIM record / mails that are already DKIM signed.
Where does this come from? From ARC signing? Maybe this has something to do with https://github.com/vstakhov/rspamd/issues/1993?
Thanks in advance!
Steps to reproduce
1.Receive a mail from google.com/Gmail
2. Check mail.log / Rspamd webui
Expected results
No error message
Actual results
rspamd[726]: <0d54f0>; proxy; lua_dkim_sign_handler: cannot load dkim key /var/mail/dkim/google.com/private.key: cannot stat private key /var/mail/dkim/google.com/private.key: No such file or directory
Debugging information
Feb 21 12:05:01 postfix/smtpd[16632]: connect from unknown[<IP_ADDRESS>]
Feb 21 12:05:01 postfix/smtpd[16632]: Anonymous TLS connection established from unknown[<IP_ADDRESS>]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
Feb 21 12:05:02 postfix/smtpd[16632]: 208422004F9FA6: client=unknown[<IP_ADDRESS>]
Feb 21 12:05:02 postfix/cleanup[16635]: 208422004F9FA6<EMAIL_ADDRESS>Feb 21 12:05:02 rspamd[717]: <8e7db3>; proxy; lua_dkim_sign_handler: cannot load dkim key /var/mail/dkim/google.com/private.key: cannot stat private key /var/mail/dkim/google.com/private.key: No such file or directory
Feb 21 12:05:02 rspamd[717]: <8e7db3>; proxy; lua_dkim_sign_handler: cannot load dkim key /var/mail/dkim/google.com/private.key: cannot stat private key /var/mail/dkim/google.com/private.key: No such file or directory
Feb 21 12:05:04 postfix/qmgr[693]: 208422004F9FA6<EMAIL_ADDRESS>size=3611, nrcpt=1 (queue active)
Feb 21 12:05:04 dovecot: lmtp(16638): Connect from local
Feb 21 12:05:04 postfix/smtpd[16632]: disconnect from unknown[<IP_ADDRESS>] ehlo=2 starttls=1 mail=1 rcpt=1 data=1 quit=1 commands=7
Feb 21 12:05:04 dovecot: lmtp(info@mydomain): CZPaEmBSjVr+QAAAWYdMWg: sieve<EMAIL_ADDRESS>stored mail into mailbox 'INBOX'

Dockerfile
FROM hardware/mailserver:1.1-latest

ENV ADD_DOMAINS=xxx

ENV DBHOST=mariadb
ENV DBNAME=postfix
ENV DBUSER=xxx
ENV DBPASS=xxx

ENV RSPAMD_PASSWORD=xxx
ENV REDIS_HOST=<IP_ADDRESS>

ENV DISABLE_GREYLISTING=true
ENV TZ=Europe/Berlin
ENV VMAILUID=5000
ENV VMAILGID=5000

COPY rootfs /

rootfs only contains some adjusted dovecot config, additional modules for icinga monitoring and adjusted ratelimit.conf for rspamd.
docker-compose.yml
  mailserver:
    build: mailserver/
    container_name: mailserver
    domainname: mydomain
    hostname: mail
    restart: on-failure:5
    cpu_shares: 256
    mem_limit: 500m
    security_opt:
      - apparmor:docker-default
    ports:
      - "25:25"
      - 143:143
      - 587:587
      - 993:993
      - 4190:4190
      - 11334:11334
    volumes:
      - /var/www/vhosts/mail.mydomain/docker:/var/mail
      - /dev/log:/dev/log
    depends_on:
      - mariadb
  redismail:
    container_name: redismail
    image: redis
    command: redis-server --appendonly yes
    restart: on-failure:5
    network_mode: service:mailserver
    cpu_shares: 128
    mem_limit: 128m
    read_only: true
    security_opt:
      - apparmor:docker-default
    volumes:
      - /var/www/vhosts/mail.mydomain/docker/redis:/data

Where does this come from? From ARC signing? Maybe this has something to do with vstakhov/rspamd#1993?

Maybe, does your mail server acts as forwarder ?
Rspamd latest stable version (1.6.6) have fixed a signing bug, it may be related to your issue :
* [Fix] Do not try to sign unknown domains

But you already use this version I presume.
hardware/mailserver 1.1-latest sha256:f3a51071cf0fac8334a119037f[...] 7 days ago

If you don't have the latest rspamd version, rebuild your custom Dockerfile.

rootfs only contains some adjusted dovecot config, additional modules for icinga monitoring and adjusted ratelimit.conf for rspamd.

Dovecot configuration and rspamd modules can be customized without having to create a custom docker image.
https://github.com/hardware/mailserver#custom-configuration-for-dovecot

Maybe, does your mail server acts as forwarder ?
No, the inbox is managed by this server, it is not even a forward to another address on this same server.

If you don't have the latest rspamd version, rebuild your custom Dockerfile.
Yes, I already have rspamd 1.6.6, it happened with older versions but also with this version.

Did you ask on the rspamd mailing list ? I can't reproduce your problem on 3 different mail servers.

No, not yet. I'll try that, for now it's just an error message, the messages are still delivered and are not marked as spam.
Thanks for your support!

I'm starting to see the same thing.
I don't have many details to provide, but I'm using the "default config" for now.

/cc @vstakhov, @fatalbanana

/cc @vstakhov, @moisseev, @fatalbanana

rspamd[726]: <0d54f0>; proxy; lua_dkim_sign_handler: cannot load dkim key /var/mail/dkim/google.com/private.key: cannot stat private key /var/mail/dkim/google.com/private.key: No such file or directory

Setting sign_local = false in dkim_signing configuration should help I suppose.

In fact, when you hide the real IP of a client, everything is expected to be broken. Don't do it, or this is a least bad problem you have...

@vstakhov Are you referring to remove the headers with the ip on sent messages or something else?
https://github.com/hardware/mailserver/blob/master/rootfs/etc/postfix/header_checks

@vstakhov https://github.com/hardware/mailserver/blob/master/rootfs/etc/postfix/main.cf#L206-L214
milter_mail_macros    = i {mail_addr} {client_addr} {client_name} {auth_authen}

@vstakhov local_addrs/local_networks Will need to add the local subnet for docker?

If you forward mail from some local MTA to another MTA that, in turn, calls Rspamd, then you need to patch the source IP based on received headers. There is no out-of-the-box functionality for that but you can do it using Lua API. local_addrs is irrelevant in this case.

In fact, when you hide the real IP of a client, everything is expected to be broken. Don't do it, or this is a least bad problem you have...

@vstakhov  thanks for the advice, I will change that.
However, header cleanup is only triggered when sending a mail using submission (587) or smtps (465) ports with a mail client, so that does not explain why rspamd, sometimes, try to sign incoming mails.
https://github.com/hardware/mailserver/blob/5554e33ef16a669dce5c01d1668b24fa1a7772c1/rootfs/etc/postfix/master.cf#L13-L29

Header cleanup? My question is why Rspamd has an email that pretends to be sent from <IP_ADDRESS>?

It seems the problem is in the network settings Docker or compatibility rspamd with docker?

pretends to be sent from <IP_ADDRESS>

Docker NAT subnet <IP_ADDRESS>/16

Oh ok, We are not talking about the same thing. I did not see the local docker address in IP address field :)
@navossoc @j-schumann I can't reproduce this with my mail server :

A always get the correct IP address in rspamd WebUI field and not a local docker address. Maybe your problem is related to this issue :
#43 - sometimes Postfix thinks <IP_ADDRESS> is the client_ip
This is a docker ~bug~ behavior. Read this : https://github.com/moby/moby/issues/15086

@vstakhov not sure who or what is spoofing that ip address, but...
<IP_ADDRESS>/16 and <IP_ADDRESS>/16 are two networks on docker.
172.18.x - http_network
172.19.x = docker_mail_network
This is the last email I have received:
Return-Path<EMAIL_ADDRESS>Delivered-To<EMAIL_ADDRESS>Received: from mail-nyc3.revenda.email
	by mail-nyc3.revenda.email (Dovecot) with LMTP id a2pjI/J1olo0SQAA7+qA6A
	for<EMAIL_ADDRESS>Fri, 09 Mar 2018 11:54:26 +0000
Authentication-Results: mail-nyc3.revenda.email
Received: from mail-qt0-x230.google.com (unknown [<IP_ADDRESS>])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mail-nyc3.revenda.email (Postfix) with ESMTPS id 1A626B5B
	for<EMAIL_ADDRESS>Fri,  9 Mar 2018 11:54:25 +0000 (UTC)
Received: by mail-qt0-x230.google.com with SMTP id a23so10376033qtn.0
        for<EMAIL_ADDRESS>Fri, 09 Mar 2018 03:54:25 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:from:date:message-id:subject:to;
        bh=5+KErOVOO3puPIHf0zTxo10UMRiowm4yep6DYJ72394=;
        b=rnVIKqdIyhbAf3TND4sET16Ca+g0CUO8aIv4WNpHrxHv/KNbmvLCMg+O1D4vJBJ2GW
         t08CNwNCrimGG6bp5pXyC7vOX3SjwrYHBY/3CcYPKyuGj7nbnEt0OVki9VW0l5szBnWk
         C2HLAUOQUQaRCxlfOWcstz4Cq4YX1uHbgKD1Uv36eRUd0QTccpVg1vkHJ/H+Ac+uaT7f
         k1cJYStu+xwBbOObW0fgtSUtzMSjd9fNu2Q3OU6qF4Pn+G5RUKIPhi0BnFR/HmHcJc2T
         mG3oGP8lnVH27VvqQ5Z4a+zofYk7NtLCfGdzfIyk/mhQKdDwZtBV39xeWvEnlggbZMuK
         dXOQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=5+KErOVOO3puPIHf0zTxo10UMRiowm4yep6DYJ72394=;
        b=WthxHK2KRyqOcsGoIa83mia5+svoLn+Ju9mnio7y2mdS65x06ysgXEK25CMA+EhHbw
         +7jsEJiJwXmM6NpK/utioj+hIh7K1eW4dLFxqykMlQI/1h0tdIjNG2JAWFWPV/I58b+d
         zZ3vzGUHyoGslOnwf5cDXdbgIF9waNjcAFvGTgelwfvF/Jn4ie4BW4kBH6MIIbwSebSn
         xYAhXWbYNwbb1knueJhVeLhRe9bU+93SJo8daCJ1XVcoST6waZKxeKyQrcKbV6NK3ktL
         Boo0mMVbAstoxZMjU2GDo3AD+OWWUg2ILwyHM6fZJElQT0gfggF/Azj4G2mdMn/6QKTH
         dR3A==
X-Gm-Message-State: AElRT7GNKs6Yv2F5Fk0OvMYj8qXsevFKomljgPE2U/SzsztZx5XmGMTh
	kXdIHSKoMfFSuNcxFPb57bqx0IQgemfDug+YeomkAw==
X-Google-Smtp-Source: AG47ELtZ/JOJmFVjLMbXIx0Dz5u4p8Ao0Ws7WlFn2nemiNdoNyy0EcYiW0UdiSC27KlPv6XWrgPPFanu3mCVSoh0dW8=
X-Received: by <IP_ADDRESS> with SMTP id z32mr44750251qtg.119.1520596465003;
 Fri, 09 Mar 2018 03:54:25 -0800 (PST)
MIME-Version: 1.0
Received: by <IP_ADDRESS> with HTTP; Fri, 9 Mar 2018 03:54:04 -0800 (PST)
From: Rafael Cossovan<EMAIL_ADDRESS>Date: Fri, 9 Mar 2018 08:54:04 -0300
Message-ID<EMAIL_ADDRESS>Subject: oi oi oi
To<EMAIL_ADDRESS>Content-Type: multipart/alternative; boundary="f4030437a1dc24afcb0566f97453"
X-Rspamd-Queue-Id: 1A626B5B
X-Rspamd-Server: mail-nyc3.revenda.email
X-Spam-Status: No, score=0.40
X-Spamd-Result: default: False [0.40 / 20.00]
 RCVD_COUNT_ONE(0.00)[3]
 RCVD_NO_TLS_LAST(0.00)[]
 FROM_HAS_DN(0.00)[]
 MIME_GOOD(-0.10)[multipart/alternative,text/plain]
 RCPT_COUNT_ONE(0.00)[1]
 TO_MATCH_ENVRCPT_ALL(0.00)[]
 FROM_EQ_ENVFROM(0.00)[]
 ARC_NA(0.00)[]
 TO_DN_NONE(0.00)[]
<EMAIL_ADDRESS> R_PARTS_DIFFER(0.50)[100.0%]

--f4030437a1dc24afcb0566f97453
Content-Type: text/plain; charset="UTF-8"

erro?

--f4030437a1dc24afcb0566f97453
Content-Type: text/html; charset="UTF-8"

erro?

--f4030437a1dc24afcb0566f97453--```

Return-Path<EMAIL_ADDRESS>Delivered-To<EMAIL_ADDRESS>Received: from mail.meshup.net
	by mail.meshup.net (Dovecot) with LMTP id 1EXdFtvDolpTIAAAlGKyKw
	for<EMAIL_ADDRESS>Fri, 09 Mar 2018 17:26:51 +0000
Authentication-Results: mail.meshup.net; dmarc=pass policy=none header.from=gmail.com; dkim=pass header.d=gmail.com; spf=pass<EMAIL_ADDRESS>Received: from mail-oi0-f44.google.com (mail-oi0-f44.google.com [<IP_ADDRESS>])
	(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
	(No client certificate requested)
	by mail.meshup.net (Postfix) with ESMTPS id A4EEC261568
	for<EMAIL_ADDRESS>Fri,  9 Mar 2018 17:26:50 +0000 (UTC)
Received: by mail-oi0-f44.google.com with SMTP id u73so7525029oie.3
        for<EMAIL_ADDRESS>Fri, 09 Mar 2018 09:26:50 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20161025;
        h=mime-version:from:date:message-id:subject:to;
        bh=m4e+dzA7RopS1WGxh5RfZ95P9rmGxG5oO2o6jZNJnUw=;
        b=U00aWnPG6zPViPEfLZb46xul7z/dxJjwc7kTobCZ+XTnR2/6E1LTAEX+JjP4GAS2fA
         nWx4fnZkQtG7QTLX4ny5sf3Ygo5VNYVt1y0fbBcf1tOyYBw67bjxZMwRRt86COPtCIjB
         QixEqzsS+HIawX6hfBAyPVsxU4mmRqLm7eKml9O8c+JlmbezE2uuoIWrG+Y6bhFonZ97
         nic7dyTypJVUWMTARF16w0dA+PU/wY1lwQZvZfMcHSAA91GoPKL/72Rozm3UeNdOP/kd
         20iKO0MNUaJGwbapcGpLE1a8HKyjbM5xoZcsx3xWxPgj+4eKgMH/KiLHWJtOKhvHi+L7
         N/8A==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20161025;
        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;
        bh=m4e+dzA7RopS1WGxh5RfZ95P9rmGxG5oO2o6jZNJnUw=;
        b=i/582wqdtT0tT/YT3VPIJbEzelMikipRuJG32P3mO1H3WSZqliG56FF6UdYg31o+ib
         +UEMRFQrtnNPW/llnt71jBz1weLmI3uQcYFXsopyDPZRJ9R4sCnTPXR0Wju3VYIf4lG6
         T63jEM2PxQYSXYvJd1LleYmsPSjeM1LIMwazwoaO8ZC61KtS9uCtlgs8J81579YBwIUE
         n64g6rDfonvLf5EfUOc9WrO6ZqY699NMc65LhpiJ6yVXSnryI4oYtQT2Nh41vI+Qid39
         CjskjZ7dgdkyBIMln2MYB/PlDW0ZZJN/O+h7dihir5/IewPtcqvS+ZiMY6Le13fEMR4g
         F39Q==
X-Gm-Message-State: AElRT7FpPzWMrKNEAvtahcRxprG6hRyt5kGyLdknljrh8+1gBi3TX6bO
	AKXXj78E26VvSNv3gZzKTGmHmAFCMSz8Zzc3chRn1QwV
X-Google-Smtp-Source: AG47ELv5NKM7Hq+Y/FAMVvmHdKOram0YKd/T79JxKG76QEv4TxA4he1S+zqRwdLGa8Qzj67f7eFCFsIouqDcLggrAxI=
X-Received: by <IP_ADDRESS> with SMTP id p15mr18883395oie.149.1520616408459;
 Fri, 09 Mar 2018 09:26:48 -0800 (PST)
MIME-Version: 1.0
Received: by <IP_ADDRESS> with HTTP; Fri, 9 Mar 2018 09:26:48 -0800 (PST)
From: John Doe<EMAIL_ADDRESS>Date: Fri, 9 Mar 2018 18:26:48 +0100
Message-ID<EMAIL_ADDRESS>Subject: test
To<EMAIL_ADDRESS>Content-Type: multipart/alternative; boundary="001a1144f150ddadc10566fe18cb"
X-Rspamd-Queue-Id: A4EEC261568
X-Rspamd-Server: mail.meshup.net
X-Spam-Status: No, score=-0.75
X-Spamd-Result: default: False [-0.75 / 20.00]
 TO_DN_NONE(0.00)[]
 IP_SCORE(0.00)[ipnet: <IP_ADDRESS>/17(-7.89), asn: 15169(-4.93), country: US(-0.92)]
 ARC_NA(0.00)[]
 R_SPF_ALLOW(-0.20)[+ip4:<IP_ADDRESS>/17]
 R_PARTS_DIFFER(0.50)[100.0%]
 ASN(0.00)[asn:15169, ipnet:<IP_ADDRESS>/17, country:US]
 MX_GOOD(-0.50)[cached: alt4.gmail-smtp-in.l.google.com]
 TAGGED_FROM(0.00)[]
 MIME_GOOD(-0.10)[multipart/alternative,text/plain]
 R_DKIM_ALLOW(-0.20)[gmail.com]
 FROM_HAS_DN(0.00)[]
<EMAIL_ADDRESS> TO_MATCH_ENVRCPT_ALL(0.00)[]
 RWL_MAILSPIKE_POSSIBLE(0.00)[<IP_ADDRESS>.rep.mailspike.net : <IP_ADDRESS>]
 RCVD_NO_TLS_LAST(0.00)[]
 RCPT_COUNT_ONE(0.00)[1]
 FROM_EQ_ENVFROM(0.00)[]
 RCVD_COUNT_ONE(0.00)[3]
 DMARC_POLICY_ALLOW(-0.25)[gmail.com,none]
 RCVD_IN_DNSWL_NONE(0.00)[<IP_ADDRESS>.list.dnswl.org : <IP_ADDRESS>]

--001a1144f150ddadc10566fe18cb
Content-Type: text/plain; charset="UTF-8"

test

--001a1144f150ddadc10566fe18cb
Content-Type: text/html; charset="UTF-8"

test

--001a1144f150ddadc10566fe18cb--

@hardware @navossoc @j-schumann Are you using Unbound/Swarm-mode or not?

Yeah, I think we have found the problem.
mail-qt0-x230.google.com. 21599 IN AAAA 2607:f8b0:400d:c0d::230
mail-oi0-f44.google.com. 21599  IN      A       <IP_ADDRESS>

Since my hostname don't have an ipv4 address, "something" (probably docker) spoofed his ip address.
@denji
I'm not using docker swarm and I'm using unbound as a local dns resolver.

I'm not using docker swarm and I'm using unbound as a local dns resolver.

Same.

Since my hostname don't have an ipv4 address, "something" (probably docker) spoofed his ip address.

And if you add an IPv4 address, does that correct the problem ?

And if you add an IPv4 address, does that correct the problem ?

Sorry, I expressed myself poorly up there.
Yes, my hostname does have a IPv4 and a IPv6 address, both reachable from internet.
What I was trying to say was:
The email server that gmail used to send me the email only has ipv6.
In your case, the server they used was one that only has ipv4.

Postfix stable release 3.3.0
Interoperability: when Postfix IPv6 and IPv4 support are both enabled, the Postfix SMTP client will now relax MX preferences and attempt to schedule similar numbers of IPv4 and IPv6 addresses. This works around mail delivery problems when a destination announces lots of primary MX addresses on IPv6, but is reachable only over IPv4 (or vice versa). The new behavior is controlled with the smtp_balance_mx_inet_protocols parameter.

@navossoc ok so this is definitely an IPv6 issue with docker : https://github.com/hardware/mailserver/issues/43#issuecomment-330809473

I had some progress with it...

For now (testing), I just dropped the http_network and recreated it with ipv6 enabled.
This was enough to assign a global ipv6 address to each container on that network.
Test 1: After that I tried to send another email and it failed miserable.
Test 2: I went to my DNS settings and changed my AAAA record to point to my mailserver container and boom it works properly.
So, it seems the problem is when docker forwards ipv6 packets from host to it's containers.
I'm not sure if this is a configuration issue with docker or it is a bug.
I'll investigate further at the weekend to see if I can get a better solution.

@hardware @navossoc @j-schumann Are you using Unbound/Swarm-mode or not?

No, nothing like that, simple docker-compose on host reachable via IPv4/v6.

I always get the correct IP address in rspamd WebUI field and not a local docker address. Maybe your problem is related to this issue : #43 - sometimes Postfix thinks <IP_ADDRESS> is the client_ip

It sure is, I opened that issue. The problem here is the bad support of IPv6 by Docker: I don't want to assign global IPv6 addresses to each container as @navossoc did because I want all traffic to go through the host to utilize one fail2ban instance for host + all containers, so I'm stuck with <IP_ADDRESS> showing up in the containers for all IPv6 requests as there is no immediate solution for this in https://github.com/moby/moby/issues/15086
My current question is: Why is rspamd even trying to sign incoming mails here, regardless if they are from a local IP or external? @vstakhov
I would understand if he tried to sign mails being forwarded to another server, but this could happen in the sending process, not on receival?

@j-schumann I gave up assigning a ipv6 for each container.
DigitalOcean provides only 16 ipv6 addresses 🤢 not a /64 block.
So, I created an ipv6_network and attached it only to the mailserver container. I also assigned a static ipv6 to it. Now I'm pointing my AAAA record to that container.
This will be my workaround until the ipv6 on docker gets better.
@hardware Postfix listen in both protocols, while dovecot only listen on ipv4.
Do you want a PR for this or you will drop whole ipv6?
As for the original issue, I believe there is nothing to be done, except perhaps if rspamd can be configured otherwise where this would not happen.
Seems that RMILTER is obsolete. I need to check the docs later.

@navossoc  You can make a PR to drop IPv6 protocol for Postfix I think. To my knowledge, it is not useful that a program inside a container to listen in IPv6, docker manages IPv6 routing outside the container. You confirm that ?

@hardware to be honest, I'm not a docker expert, so you may want a second opinion.
AFAIK docker is good enough with ipv4, but ipv6 support is kind 😐 yet.
If you application inside the container don't listen on ipv6, it can't be reached from the internet directly.
IMHO: To be safe is better to have both, since ipv6 is a trending and the support will get better eventually.
In my case, I'm planning to listen on both protocol and expose only the mailserver container to a ipv6 network.
So this way I can support users connecting with ipv4 and ipv6 to postfix/dovecot.
@j-schumann do you have any opinion on that?

@navossoc In my opinion every application running in a container that can be exposed via a port should support v4 & v6 for future compatibility. If it's only internal communication within the container it can be v4 only or via sockets.

As for the original issue, I believe there is nothing to be done, except perhaps if rspamd can be configured otherwise where this would not happen.

The only relevant parameter that I see is to switch sign_local to false in rspamd configuration.

@j-schumann I found this today:
https://github.com/robbertkl/docker-ipv6nat
Looks promising. I'll give a try later.
Maybe it solve our problems?

@navossoc looks interesting, one issue there said it can work with ufw, this is good. Don't know if this means it also works with fail2ban. Let me know how it goes!

@j-schumann works fine and also seems to solve the problem.
I did the same thing as before, but this time I created a ipv6nat_network and assigned it to mailserver.
Then created the docker-ipv6nat container and reverted my AAAA record back to my main ipv6.

@navossoc I got it working with my docker-compose.yml, below is an extract, important is the listing of the custom network for each service, not only the mailserver or he can't communicate with MySQL etc.
This definitely solves the issue, mailserver and other containers now see the real IPv6 address.
But it still has the same problem as docker has by default: it ignores any other iptables rules from ufw or fail2ban, so i'll have to finally ditch ipv6nat, set DOCKER_OPTS="--iptables=false" and use custom rules.
docker-compose.yml, builds the mailserver with custom dockerfile in the "mailserver/" dir:
version: '2.1'
services:

  ipv6nat:
    image: robbertkl/ipv6nat
    restart: on-failure:5
    privileged: true
    network_mode: host
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

 mailserver:
    build: mailserver/
    container_name: mailserver
    domainname: abc
    hostname: mail
    restart: on-failure:5
    cpu_shares: 256
    mem_limit: 500m
    networks:
      custom:
    security_opt:
      - apparmor:docker-default
    ports:
      - "25:25"
      - 143:143
      - 587:587
      - 993:993
      - 4190:4190
      - 11334:11334
    volumes:
      - /var/www/vhosts/abc/docker:/var/mail
      - /dev/log:/dev/log
    depends_on:
      - mariadb

  redismail:
    container_name: redismail
    image: redis
    command: redis-server --appendonly yes
    restart: on-failure:5
    network_mode: service:mailserver
    cpu_shares: 128
    mem_limit: 128m
    read_only: true
    security_opt:
      - apparmor:docker-default
    volumes:
      - /var/www/vhosts/abc/docker/redis:/data

networks:
  custom:
    driver: bridge
    enable_ipv6: true
    ipam:
      driver: default
      config:
      - subnet: fd00:dead:beef::/48

@hardware this looks like a good fix to me. Let's wait @j-schumann reply with his results and I can help to draft a wiki page explaining how to setup ipv6. I just need someone later to review it.

No problem, your solution looks great.

@hardware Yes, I can, probably wednesday I'll let you know about the results.

@hardware I did some additional tests with the 1.7.x version, but I still need to use the ipv6nat to get the correct ipv6 address of the remote sender.
So in my opinion the best way to deal with this right now (until docker fixes it) is using the ipv6nat or assigning one ipv6 address to each container, besides that, everything else seems ok.

@navossoc is that mean if I update my image to the latest version of "hardware/mailserver" I need to update the docker-compose file, too for ipv6?  Even though I don't use it?

This ipv6 problem already exists in the previous version and continues in this new version as well.
(it's a docker issue to be more specific)
If you do not have any AAAA (ipv6) entry in your DNS, then you will not get any ipv6 connection and do not have to worry about it.
Just ignore this.

Someone can make a pull request to explain ipv6nat solution ?

@hardware Sure, give me a few hours. I'll make it available tonight.

I've created a draft of the pull request, it's enough to make it work.
Feel free to edit as you wish.
Let me know if you have any doubts.
[]'s

Solution : https://github.com/hardware/mailserver#ipv6-support (thanks @navossoc).


Add upgrade_type attribute to cluster resource

upgrade_type is now added for BASIC and STANDARD clusters.  It can be used to dictate how the cluster handles major version upgrades.  Valid values for upgrade_type are AUTOMATIC and MANUAL.  BASIC clusters support only AUTOMATIC upgrades.
Commit checklist

[x] Changelog
[x] Doc gen (make generate)
[ ] Integration test(s)
[x] Acceptance test(s)
[x] Example(s)

@mdlinville I forgot to add you to this PR for docs review.  Can you take a look here?  I've already merged it down but I can send a follow up with any suggestions you have.


missing schema

Hi janfitz!
I like your project's simplicity, I could use it for my own, but I haven't found any schema info. I keep getting
SQLite3::query(): Unable to prepare statement: 1, no such table: car ... errors. Could you help me in this?
Thank you!

Hi r3sist,
thank you for your message. In master you can now find two example sqlite databases. Everything should works fine. You can clone the scheme or edit one of included database. It should be enough to pull changes. If some error occured, please let me know.

Works perfect! Thank you for your quick help!


Tuple schema discrepancy with Dataframes

Running into some issues when working with Tuples and Dataframes. Constructing a Dataset with a tuple of conjunctiontopics: (Option[String], Option[String], Option[String])  for my case class. The C* schema is conjunctiontopics frozen<tuple<text, text, text>>,. The Dataset schema inferred from the cassandra table is different from the dataset. When I run newDS1.printSchema I see
 |-- conjunctiontopics: struct (nullable = true)
 |    |-- _1: string (nullable = true)
 |    |-- _2: string (nullable = true)
 |    |-- _3: string (nullable = true)

The upstream cassandra table returns
|-- conjunctiontopics: struct (nullable = true)
 |    |-- 0: string (nullable = true)
 |    |-- 1: string (nullable = true)
 |    |-- 2: string (nullable = true)

This results in the following error when I attempt to run a join on both datasets.
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '(`conjunctiontopics` = `conjunctiontopics`)' due to data type mismatch: differing types in '(`conjunctiontopics` = `conjunctiontopics`)' (struct<_1:string,_2:string,_3:string> and struct<0:string,1:string,2:string>).;; 'Project [pipelinekey#143, placeid#147, periodtype#141, period#142, externalsourceid#139, conjunctiontopics#150, periodstartdate#138L, periodenddate#140L, mentioncount#144, avgpossentiment#145, avgnegsentiment#146, centroidlat#148, centroidlon#149, mentioncount#84L, periodstartdate#85, periodenddate#86, centroidlat#87, centroidlon#88, avgnegsentiment#89, avgpossentiment#90, insertiontime#91]

This project is no longer using GitHub issues; instead please use JIRA:
https://github.com/datastax/spark-cassandra-connector#reporting-bugs


[CI] Initialization of e2e integration tests

Description
This PR is incomplete, but uses a common approach that other adapters are using to perform end-to-end testing.

@jbyers19 you might make quick work of the expected pods and appropriate test assertions.

@leecalcote, good stuff. I'll need a bit to grok the tests before making changes. Did you want to merge this PR now with the basic test structure there? Or leave it as WIP until I can update the assertions?

@jbyers19, oops, I missed your question. Originally, I think that this PR can wait around under you validate the appropriate assertions. @Revolyssup might have carried this work forward in another PR, though (and might have engaged you, I don't remember). @Revolyssup, should we close this PR?

@jbyers19, oops, I missed your question. Originally, I think that this PR can wait around under you validate the appropriate assertions. @Revolyssup might have carried this work forward in another PR, though (and might have engaged you, I don't remember). @Revolyssup, should we close this PR?

What do you guys think?

@leecalcote  Yes I think we can close


Add ruby 3.3 to test matrix and remove EOL versions

Thanks for the PR! Sorry for the long delay with merging this - I'll push it in then bump up the patch versions to latest separately.

Wrapped up in passing under the v2.2.0 tag and pushed to https://rubygems.org/gems/omniauth-azure-activedirectory-v2, but of course the test stuff doesn't have an impact on the published gem.


Fixed EfficientNetV2 b parameter not increasing with each block.

Closes #16144
It was found that the survival_probability of Efficientnet V2 networks was always 0.0, where it should be slowly increasing with each block size. The reason was I forgot to increase b parameter in those networks.
Thanks @SergioG-M for the find.

@fchollet I just checked and the weights seem to be loading correctly. The only difference in the models is that, there are more Dropout layers, no new variables have been introduced.


Add Ops.hlo_call(::String, args...)

idea: we could hash the code and just the call the function if a function with the same hash is already present in the module (preventing duplicate functions).
So that:
for i in 1:N
  x = Ops.hlo_call(code, x)
end

has the function just once (but multiple calls).

that can be done directly with a Dict{HLOIR,Function} (or sth like that) and then calling ```julia
get!(dict, code) do code
compile(code)
end

that can be done directly with a Dict{HLOIR,Function} (or sth like that) and then calling

ah right, do we have a place to store such state within a single trace ? Otherwise my idea is to name these function like main_$(hash(code)) and see if it exists within the module before adding it.

I have a question... What is hlo_call returning? A function? Wouldn't it be better to just inject the MLIR code in the current MLIR block?

No, it is returning a tuple of tracedarray returned by the call of the function.
i.e. Reactant.@jit Ops.hlo_call(code, x, y) is a tuple of concrete arrays.

So hlo_call currently returns a tuple of arrays (one for each result), should we special case nresults() == 1 to return the only array (no tuple) ?

So hlo_call currently returns a tuple of arrays (one for each result), should we special case nresults() == 1 to return the only array (no tuple) ?

Honestly I think it’s better to always return the tuple. That way folks using it don’t need to special case if there are multiple returns or not

this is amazing and opens a way to do things like #354


Problem at Storage::file();

I cannot get any files from this function if I specify a directory even though the dir name is correct. I can see the files if I use allFiles() bun I need only the files in the current dir to be diplayed.
I need to say that this worked before today as I've seen an update to the filesystem drive. Most certein todays update broke the method.
$files = Storage::files('seeders');
dd($files);

// returns empty array

Which update are you referring to? Which versions? There is no file()/files() method in Filesystemmanager

Please ask on the forums. :)

Graham, can we get this opened back up?
This functionality is documented on the 5.1 documentation at http://laravel.com/docs/5.1/filesystem#directories
After doing some research, it looks like it only will retrieve files from approot\storage\app and it's subdirectories.
This is in contrast to the http://laravel.com/docs/5.1/filesystem#storing-files  which presently appears to take any valid path on the file system.
It's either an issue that should be addressed in the files command, or the documentation should be updated so dev's know that it's limited to the storage/app directory and it's children. Thanks.

Jack

+1 It's a real documentation or Storage issue.

https://github.com/laravel/framework/blob/5.1/src/Illuminate/Filesystem/FilesystemAdapter.php#L221

You should pass the whole directory path here.

Can we get real confirmation on this instead of locking the conversation?

Relevant documentation:

When using the local driver, note that all file operations are relative to the root directory defined in your configuration file. By default, this value is set to the storage/app directory. Therefore, the following method would store a file in storage/app/file.txt:

Storage::disk('local')->put('file.txt', 'Contents');

http://laravel.com/docs/5.1/filesystem

It's a real documentation reader issue :)
The documentation permit us to get to the essential part, allowing us to bypass the mandatory.
Laravel is so intuitive that I assumed I was able to browse my whole file system (or at least my whole project tree) as I would do with a native php methods wrapper and it leads @basementjack and I to a misunderstanding empty array without any error (obviously).
Maybe the doc could remind us that we use a specific disk and driver but it may quickly escalate to an idiotic situation such as "of course, do not forget to open you php tags"...
Maybe the documentation could teach us, impatient developers, to take enough time to check the fundamental but, honestly, good luck with that.
Today I learned my experience does not allow me to ignore the “getting started” part and I had to disturb @GrahamCampbell and @taylorotwell in order to discover this. So sorry about this.
I wish we won't be that numerous to use that way before understanding Storage class, and I'd like to use this useless issue comment in order to thank you guys for the Laravel framework.
Sorry for the noise.

Don't worry about it! Not a big deal :)
On Sat, Oct 10, 2015 at 6:29 PM, Laurent Goussard
<EMAIL_ADDRESS>wrote:

It's a real documentation reader issue :)
The documentation permit us to get to the essential part, allowing us to bypass the mandatory.
Laravel is so intuitive that I assumed I was able to browse my whole file system (or at least my whole project tree) as I would do with a native php methods wrapper and it leads @basementjack and I to a misunderstanding empty array without any error (obviously).
Maybe the doc could remind us that we use a specific disk and driver but it may quickly escalate to an idiotic situation such as "of course, do not forget to open you php tags"...
Maybe the documentation could teach us, impatient developers, to take enough time to check the fundamental but, honestly, good luck with that.
Today I learned my experience does not allow me to ignore the “getting started” part and I had to disturb @GrahamCampbell and @taylorotwell in order to discover this. So sorry about this.
I wish we won't be that numerous to use that way before understanding Storage class, and I'd like to use this useless issue comment in order to thank you guys for the Laravel framework.
Sorry for the noise.
Reply to this email directly or view it on GitHub:
https://github.com/laravel/framework/issues/9572#issuecomment-147136336


Android tests are failing due to missing class

https://circleci.com/gh/facebook/react-native/496
[x86] Install        : libreactnativejni.so => /home/ubuntu/react-native/ReactAndroid/build/react-ndk/all/x86/libreactnativejni.so
make: Leaving directory `/home/ubuntu/react-native/ReactAndroid/src/main/jni/react/jni'
:ReactAndroid:packageReactNdkLibs
:ReactAndroid:compileDebugJavaWithJavac/home/ubuntu/react-native/ReactAndroid/src/main/java/com/facebook/react/views/webview/ReactWebViewManager.java:32: error: package com.facebook.react.common.build does not exist
import com.facebook.react.common.build.ReactBuildConfig;
                                      ^
/home/ubuntu/react-native/ReactAndroid/src/main/java/com/facebook/react/views/webview/ReactWebViewManager.java:78: error: cannot find symbol
    if (ReactBuildConfig.DEBUG && Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT) {
        ^
  symbol:   variable ReactBuildConfig
  location: class ReactWebViewManager

cc @mkonicek (probably need to add ReactBuildConfig for https://github.com/facebook/react-native/commit/8cd7730080976deeb9a94046717b0cbe32e494f7)

Also, this code runs on whatever thread loads the class but WebView.setWebContentsDebuggingEnabled needs to run on the UI thread so moving this out of a static initializer might make sense too.

Thanks for keeping an eye on it. My commit broke it. Fixed here: https://github.com/facebook/react-native/commit/a3d16c0178d3794991f1cb2f656b66078efa2a26

Thanks Martin!


Prepare for Debug Logging v2

Adjust to the new debug logging message format implemented in https://github.com/deltachat/deltachat-core-rust/pull/3958

@faassen, as the debug logging pr has been merged, it would be nice to update this repo to reflect the changes.


Update mergeConfigFrom key.

The key was set to skeleton, which is the default value in https://github.com/spatie/skeleton-php. I updated it to reference the actual config file in this repository.

Nice catch!


Add script to clean dev environment

Script that the Elyra dev team has shared among themselves
and that should be useful for others as well.
Developer's Certificate of Origin 1.1
   By making a contribution to this project, I certify that:

   (a) The contribution was created in whole or in part by me and I
       have the right to submit it under the Apache License 2.0; or

   (b) The contribution is based upon previous work that, to the best
       of my knowledge, is covered under an appropriate open source
       license and I have the right under that license to submit that
       work with modifications, whether created in whole or in part
       by me, under the same open source license (unless I am
       permitted to submit under a different license), as indicated
       in the file; or

   (c) The contribution was provided directly to me by some other
       person who certified (a), (b) or (c) and I have not modified
       it.

   (d) I understand and agree that this project and the contribution
       are public and that a record of the contribution (including all
       personal information I submit with it, including my sign-off) is
       maintained indefinitely and may be redistributed consistent with
       this project or the open source license(s) involved.

I pushed an update addressing all my comments. It updates the script to default to use Lab 2.0 and removes some unnecessary parts like the nbdime/git install since those would be properly handled by elyra when using the latest versions.

I see, so not handling this on the script, but really leaving it to be handled during the elyra install. That works.

After removing the whole installation piece, we now get:
pip install --upgrade elyra

Installing collected packages: nbdime, jupyterlab, jupyterlab-git, elyra
  Found existing installation: jupyterlab 2.1.1
    Uninstalling jupyterlab-2.1.1:
      Successfully uninstalled jupyterlab-2.1.1
Successfully installed elyra-0.10.0 jupyterlab-1.2.14 jupyterlab-git-0.10.1 nbdime-1.1.0

pip install --pre --upgrade elyra

Installing collected packages: nbdime, jupyterlab-git, elyra
Successfully installed elyra-0.11.0rc0 jupyterlab-git-0.20.0rc0 nbdime-2.0.0


Surface scaffolding for Blazor apps

Before working #15651 (WRT https://github.com/dotnet/aspnetcore/issues/18747) ...

Add a short section describing the use of the scaffolder with a Blazor Server app. I'll confirm basic behavior and call out any gotchas.
Cross-link this section from the Blazor Auth topic.

We'll pick back up with further updates on #15651 after I receive notes from the PU (or they'll take it directly).

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 804e24dd-7e30-8cf5-2c99-1ffacb4582f6
Version Independent ID: 3e25547b-e2c9-6634-2c38-2a77207f17d4
Content: Scaffold Identity in ASP.NET Core projects
Content Source: aspnetcore/security/authentication/scaffold-identity.md
Product: aspnet-core
Technology: aspnetcore-security
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

Closing as a dup of https://github.com/aspnet/AspNetCore.Docs/issues/15651. Will take it up further on that issue.


Android OpenCL tutorial

description for android tutorial4

cool! :+1:

@apavlenko Please squash commits into one (to eliminate unnecessary intermediate changes)

Makes sense. Done.
(just to mention: this PR has relation to #4239)

Thanks!
:+1:


Cant join airdrop error 4207

Wondering if anyone knows a fix for this. I have been trying to join the airdrop since it was announced. I have 3 devices and 3 private keys and I get all 3 green checkmarks. When i click "become airdropee" it says "sorry you are not able to join airdrop"
I dont know what to do next, I tried reinstalling the app and trying a different wifi signal incase i was blocked for some reason.
I just want to get the airdrop man...

@maxtaco any reason why all of these related issues keep getting closed?


ENH: Corrected imaginary PLV #6851

ENH: #6851
Implements the corrected imaginary PLV  (ciPLV)

... and also make sure it's tested in mne/connectivity/tests/test_spectral.py like the other methods

@AdoNunes let me know when to look again

Example looks reasonable:
https://15861-1301584-gh.circle-artifacts.com/0/dev/auto_examples/connectivity/plot_mne_inverse_label_connectivity.html
Will merge once travis comes back happy, thanks @AdoNunes !

Ran style locally and Azure is happy, so I'll merge


Move rcl structs to end of Node declaration

Cherry-picked from #342, should help towards #369.
CC @maspe36

I think with this we can close out https://github.com/ros2-rust/ros2_rust/issues/369, https://github.com/ros2-rust/ros2_rust/issues/366, https://github.com/ros2-rust/ros2_rust/issues/365, and https://github.com/ros2-rust/ros2_rust/issues/367.
If we see an intermittent failure again we can create more appropriate issues. Thanks for your work @luca-della-vedova!


Dapp-indexer bug

issue:
Observed bugs related to dapp and indexer

If contracts does not have a yielder or offseter address, farming routes we panicking.
Add Option to yielder_address and offseter_address


Understanding +page.server.ts & +page.svelte behaviour.

Hi,
I wonder if you might be able to help me work out what is going on.
I have partitioned a path (/kit) of my static sveltekit app to use your edge-bundled server through an origin-request handler.
I have disabled caching for this behaviour.
In the root layout of /kit I have set prerendered to false.
So, I have an oauth callback:
/kit/accounts/oauth/callback?code=xxxx&......
I have a load function in a +page.server.ts file
export async function load({ url }) {
  return { url: JSON.stringify(url) }
}

and if I display the data on the page I get the cloudfront url (i.e not my custom domain), without querystring.
(So for example, on the server I am trying to exchange the code for an oauth token, but I cannot read the code from the querystring because it is empty)
Can you help me to rationalise the data flow of what is happening when the oauth callback hits the cloudfront which then forwards to the lambda edge bundled on the viewer-request handler and what steps I might have missed or architectural floors I have not considered to allow this to work.
Thank you in advance.

There is a note in the svelte docs:
you may need to configure the adapter in order for the URL to be correct
https://kit.svelte.dev/docs/load#using-url-data
I've also explicitely told cloudfront to forward host header and querystring

I needed to add an origin request policy to my distribution.
https://docs.aws.amazon.com/cloudfront/latest/APIReference/API_OriginRequestPolicy.html


generic validator and dynamic elements

Hi @DeborahK the genericValidator looks super cool, and I was keen to use this. However while doing a deep dive in the code, I noticed that you subscribe in an ngAfterViewInit method, and in it we fetch all the controls using a @ViewChildren to get the form elements and also subscribe to their Blur Events observable.
If we now have a Form Array and are adding extra tags, then as the component class is already created, our formInputElements will not get updated, and we will not be doing any validation on the newly added Form Controls, and so they will miss out on being validated by the genericValidator.
It may be possible to unsubscribe and then resubscribe each time you add a control, but this didn't feel ideal either.
Anyhow, this may be a possible gotcha, and therefore perhaps worth pointing out in the Read Me.

The basic code in the generic validator does not support FormArrays. It has no facility for tracking array elements and associating an error message with a specific instance of a control in the array.


i need help on the source coding

Describe your problem here. If you are modding, please try testing your problem in a clean version of the engine instead, this ensures the problem is actually caused by the engine itself. Also, if you're using an older version of JSE, please try the latest action build in the Actions tab or the latest release build. The issue has probably been fixed since that version. Oh yeah, and also be sure to check the pinned 'JS Engine: Known Issues' issue, to double check if your issue hasn't already been found!

how do i fix this
Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build?


Use ToPyObject instead of GetPropertyValue for #[pyo3(get)]

Some how it works.
cc: #903

I think it might not work for T: PyClass ? A test must be missing.

Ah, you're right. Thank you.
Maybe we need another way to enable get without Clone

See my comment in #803 about Clone for PyObject... I'm excited to see what you think of it!


Add support for specifying barcode size

Support specifying the size of the barcode in templates.
Internal reference URM-157550

We've released version, v0.7.0.
Two new fields, "Barcode width" and "Barcode height", may be set in template configuration. If "Show as barcode" is checked, these two fields may be edited (just like the existing "Barcode encoding" field).
If values are not provided, the "Barcode width" will be set to 1 and the "Barcode height" will be set to 20 when the barcode is printed.
If "Show barcode value" is checked, the barcode value will now print in the selected font size.
If you test this new feature, please add feedback/comments to this issue.  Thank you!


PP-2655 Restrict demo and prototype payments to sandbox

PP-2655 Restrict demo and prototype payments to sandbox
What

Restricted demo and prototype payments to sandbox accounts
Created restrict_to_sandbox middleware
Added tests for new middleware

The middleware test suite has a negative test. I was unsure as to whether we should add it to each controller or not. Can do, its no biggy


appaloosa, flavobacterium

This pull request was generated by the 'mq' tool
[test]
flake rate: 0.1
logical conflict every: 1000
sleep for: 2100s
close stale after: 24 hours
[pullrequest]
requests per hour: 100
deps=[a,f]

/trunk merge


Print match only

Is it possible to only print the match?
Say I have the following file:
Lorem Ipsum is simply dummy text of the printing and typesetting industry.

If I run ag "L.*m" is there a way to have it only print Lorem and not the entire line?

Closable now?

First of all, thanks for your work @gurgeous!
However, I am still having some issues with the new option. All examples assume we have a file foo.txt with the following contents in the current directory:
foo foobar
bar
baz
foobar
quux

(a) It would be good to add the -o option to the man page. Potentially there are more options missing, but I don’t have the time to check right now.
(b) One big issue for me is when multiple matches occur on the same line. Consider the output of the pattern 'f\w+' with my workaround presented above:
~$ ag --color 'f\w+' | ruby -ne 'puts $_.scan(/\x1b\[30;43m(.*?)\x1b/)'
foo
foobar
foobar

However, using the -o option this yields
~$ ag -o 'f\w+'
foo.txt
1:foofoobar
4:foobar

Notice how the matches are joined together. This makes it impossible to use this for proper scripting. I think this calls for another option --multi-match that works a bit like --vimgrep. It would enable printing multiple matches on multiple lines and also enable the --column option by default. This would make it re-usable as part of --vimgrep. The --column option should be enabled by --multi-match. The --multi-match option should then be automatically enabled when using -o. Some test cases follow:
The --multi-match option should enable --column automatically:
~$ ag --multi-match 'f\w+'
foo.txt
1:1:foo foobar
1:5:foo foobar
4:1:foobar

The --multi-match option should work with --color (colored output highlighted with [...]):
~$ ag --multi-match 'f\w+'
foo.txt
1:1:[foo] foobar
1:5:foo [foobar]
4:1:foobar

The -o option should enable --multi-match automatically:
~$ ag -o 'f\w+'
foo.txt
1:1:foo
1:5:foobar
4:1:foobar

The --vimgrep option should be equivalent to --multi-match --nogroup. Probably we could replace the current --vimgrep implementation by simply setting --multi-match --nogroup:
~$ ag --vimgrep 'f\w+'
foo.txt:1:1:foo foobar
foo.txt:1:5:foo foobar
foo.txt:4:1:foobar

~$ ag --multi-match --nogroup 'f\w+'
foo.txt:1:1:foo foobar
foo.txt:1:5:foo foobar
foo.txt:4:1:foobar

(c) As I plan on using this for scripting, I need to extract the matches, and the matches only from the output. I know I can use --nogroup and then pipe the output through cut in order to extract the matches:

Note that in the examples below, I assume the currently not implemented --multi-match behaviour that I described above, including the implied output of the column number.

~$ ag --nogroup -o 'f\w+' | cut -d: -f4
foo
foobar
foobar

Still, I think it would be much prettier and less fragile if we could configure all parts of the output. Some parts are already there, for example the --no-numbers option, so only the file name and the column number would remain:
~$ ag --nogroup --no-numbers -o 'f\w+'
foo.txt:1:foo
foo.txt:5:foobar
foo.txt:1:foobar

therefore, we would also need --nocolumn and --nofilename options:
~$ ag --nogroup --nocolumn --no-numbers -o 'f\w+'
foo.txt:foo
foo.txt:foobar
foo.txt:foobar

~$ ag --nogroup --nocolumn --nofile --no-numbers -o 'f\w+'
foo
foobar
foobar

maybe we could even have a --nolocation option that omits this kind of output completely:
~$ ag --nogroup --nolocation -o 'f\w+'
foofoobar
foobar

(e) As I think of it, the --line-numbers and --no-numbers option are named inconsistently. Shouldn't they be called --[no]numbers, just like --[no]heading, --[no]break and --[no]group? I'd keep the oddly named cousins around but print a deprecation warning to stderr, then eventually remove them in the future.
Let me know what you guys think of all this. Of course we probably need to extract some separate issues here, but I thought it would be smarter to discuss the concept first.

Side note: Ack has the --nocolumn as well as the -H, --with-filename and the -h, --no-filename options. So folks looking for a solution for scripting: ack -ho PATTERN will do the trick. Some examples for comparison:
~$ ack -o 'f\w+'
foo
foobar
foobar

~$ ack -o --with-filename 'f\w+'
foo.txt
1:foo
1:foobar
4:foobar

A few comments:
Yes, this should be in the man page. Oops. Want to submit a pull request? I think maybe you just have to edit ag.1.md.
The current implementation is very simple and just spits out matches and newlines. It would probably be better to move to something more like grep's approach, where each match gets a new line. That decision is up to @ggreer, though.
$ grep foo foo.txt
foo foobar
foobar

$ grep -o foo foo.txt
foo
foo
foo

# note line numbers
$ grep -on foo foo.txt
1:foo
foo
4:foo

Don't be surprised if this languishes until some enterprising person submits a pull request. Incremental pull requests with tests and comments are always appreciated! :)
Separately, it would be nice to review the output flags and make a few adjustments. Probably not in the context of this issue, though.

Alright, I've cleaned up the output of --only-matching in master. By default, it now behaves like grep and ack. ag -o file_to_seach.txt prints out matches on their own lines, and nothing else. If you don't specify the file to search, ag will print out file names and line numbers just like normal, though each match will still be on its own line. Except for the highlight colors, this output is identical to ack.

@ggreer thanks this looks like a good default. However it would be useful to have this work properly with --column. Currently, the output is
~$ ag -o --column 'f\w+'
foo.txt
1:1:foo
1:foobar
4:1:foobar

Notice that the column number is missing in the second line, i.e. the second match in the first line. The output should be
~$ ag -o --column 'f\w+'
foo.txt
1:1:foo
1:5:foobar
4:1:foobar

Good catch. I added a test case and fixed it in master.

Great so I could improve my original snippet from this scary monster
ag --color PATTERN | ruby -ne 'puts $_.scan(/\x1b\[30;43m(.*?)\x1b/)'

to this
ag -o --nogroup PATTERN | cut -d: -f3

Much better. Thanks for your effort. I will submit a PR containing the missing documentation (man page). The formatting options I suggested are a different issue. I think this ticket can be closed now.

Awesome! Glad this was added :thumbs up:. I'll leave it to @ggreer to close the issue when all is done with the man page update.

@zachlatta the man page was already updated, I just forgot to mention this ticket in my PR.

Ah, gotcha. So this is ready for closing then?
--
Zach Latta
http://zachlatta.com
https://github.com/zachlatta
https://twitter.com/zachlatta
On Tue, Jan 20, 2015, at 04:36 AM, Patrick Oscity wrote:

@zachlatta[https://github.com/zachlatta] the man page was already updated, I just forgot to mention this ticket in my PR.

—
Reply to this email directly or view it on GitHub[https://github.com/ggreer/the_silver_searcher/issues/400#issuecomment-70647099].

If you're happy with it – go ahead!
Cheers

Nice work, everyone. :)

What come people really want is print only a capture, not the whole match. This is already possible through the PCRE library, but it could be hard to realize how to do this with a deep dive into the documentation. A short example added to the ag documentation could help.  For example, let's say you have a bunch of  HTML anchor tags that link to https:// domains you want to pull out the domain names. This works:
  ag -o 'https://\K.*?(?=")'

The key things to know about PCREs here are:

\K is used to reset the match start, so text before it won't be returned in the match.
(?=) is the Lookahead Assesrtion construct. It looks ahead to see if the remaining text would appear, without including the text as part of the match.

By combining these two PCRE features, you can remove both preceding and following text from the match.


Use builder pattern to construct WebRtcSocket

Had a go at creating a builder for the WebRtcSocket to replace WebRtcSocketConfig, this should make things a touch easier when it comes to constructing sockets with numerous channels.
Also added a helper to the ggrs feature to allow users to more easily add an appropriate channel.

Please add WebRtcSocket::builder() -> WebRtcSocketBuilder

Good idea! Have used the same arguments as WebRtcSocketBuilder::new, but not sure this feels quite right

Look good now :) I only had a doc nit and a question.

Oh, I'd still like some dead-simple convenience methods that doesn't involve creating a builder.
let socket = WebRtcSocket::new_unreliable("wss://example.com");

As opposed to:
let socket = WebRtcSocket::builder("wss://example.com")
    .add_unreliable_channel()
    .build();

Internally, it would just call the builder.
I want to keep simple/common usage really simple as far as possible.

Oh, I'd still like some dead-simple convenience methods that doesn't involve creating a builder.

Done! Do you think we should use these or the builder in the examples?

Oh, I'd still like some dead-simple convenience methods that doesn't involve creating a builder.

Done! Do you think we should use these or the builder in the examples?

Hmmm.. I think it would be good to have at least one example that shows how to set up multiple channels, but in our current examples i think the simple versions are probably better.

Hmmm.. I think it would be good to have at least one example that shows how to set up multiple channels, but in our current examples i think the simple versions are probably better.

That seems reasonable to me, a mutli-channel example would definitely be good to have

Not really convinced, wouldn't we have to add a lot of API: ip/hostname, port, which ws and wss, path and query string building? Only to convert it to a string since that's what the browser/async_tungstenite wants anyway... I think we should just wrap WsErr::InvalidUrl (and the native variant) into one of our errors when connecting.

Ok, I think it clicked for me when you said "that's what the browser wants anyways".
Fair point.


Create Advanced CRUD Api Test Box Component

Description
Using the current CRUD test box component. Create an advanced version that can offer more customization to the requests.

@Curtis-Thomas would you mind if I add some comments to the existing code as well

@Curtis-Thomas is it ok with you if you can assign this issue in my name.

@Curtis-Thomas would you mind if I add some comments to the existing code as well

Raise an issue for this :)

@Curtis-Thomas is it ok with you if you can assign this issue in my name.

For the advanced version, could it be made as a whole new component with a different name? Then the relevant component can be inserted whenever it is needed.

@Curtis-Thomas would you mind if I add some comments to the existing code as well

Raise an issue for this :)

Thank you.. I have raised new issue for this.
https://github.com/Curtis-Thomas/free-api-list-with-react/issues/24


mp4 files do not stream

Is qtfaststart or something equivalent being run on mp4 files because they don't play until completely downloaded. webm files don't have this issue.

Similar to #21


Prompt gets garbled when fixing and need to pick for multiple imports

If I am working on a file that needs a lot of new modules fixed, and I run fix imports on it for the first time, and multiple modules have multiple matches, the prompt gets a bit garbled. Here's an example:
ImportJS: Imported `react` (0.0s)
ImportJS: Pick JS module to import for 'Foo':
1: ../../foo/foo
2: ./Foo
Type number and <Enter> or click with mouse (empty cancels): 2ImportJS: Pick JS module to import for 'Bar':
1: ../../apps/bar
2: ../../foo/bar

Notice the line: Type number and <Enter> or click with mouse (empty cancels): 2ImportJS: Pick JS module to import for 'Bar':.

Interesting. I'm assuming the prompt gets more garbled as you go along, picking modules to import (since that's handled synchronously).

We should move this issue to the vim-import-js repo

This issue was moved to Galooshi/vim-import-js#2


[WIP] KAFKA-12899: Support --bootstrap-server in ReplicaVerificationTool

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Looks good to me. Will wait for the unit test to be added.

@guozhangwang Yes, I am currently working on it. And could you kindly have a look at the voting thread? As of present, +2 bindings with +2 non-bindings. One more binding is required to be passed.

Here it is. With rebasing onto the lastest trunk, I added unit tests that verify the command line parsing.
Also, for this KIP is not passed yet (+2 bindings), I prefiexed the title with '[WIP]'.

@tombentley Could you kindly have a look when you are free? :bowing_man:

Rebased onto the latest trunk.

Hi @tombentley
Here is the update. I added additional tests (ReplicaVerificationToolTest#{testExitWithMultipleBrokerLists, testExitWithMultipleBootstrapServers), Add descriptions to exising test, with changing parameter descriptions.

Hi @tombentley,
Sorry for missing your comment on KIP-629. Reviewing KAFKA-10201, I found that it is already under progress with KAFKA-10589. So, it would be better to address this issue in this PR.
I also rebased onto the latest trunk. Thank you for reviewing! :bowing_man:

@dongjinleekr @tombentley if we intend to include this fix to 3.0 we'll need to merge asap. Please let me know so I can update the status of https://issues.apache.org/jira/browse/KAFKA-12899 accordingly

@kkonstantine It would be really great for me if this PR is included in 3.0, but it still needs one more binding +1. (It is also why it is prefixed with '[WIP]'.) Could you have a look at the voting thread and cast a vote? If so, it may be included in the next release.
Besides, I think we should merge KAFKA-10589 before this one; you know, KIP-629 is already passed.

@showuon Sorry for being late. Rebased onto the latest trunk, resolving the issues.

I think you can close this vote now since I'm also +1 binding now. :)

OMG, What an unexpected present! 😆 But could you kindly cast your vote on the mailing thread? As far as I know, the voting is only valid within the mailing list. :pray:

Closes this PR, since KAFKA-12899 has been replaced by KAFKA-17073.
Thank you to everyone who participated in the discussion. Let's continue on KAFKA-17073! :bowing_man:


Add French(fr) locale support

Hi, just a little patch for French support.

Many thanks! I've just released a new version.


RETICULATE_PYTHON env variable failed to be set

I have been following this tutorial to upload my own shiny app to shinyapps.io and I cannot get around some warnings:
Warning: The reticulate package is installed but no Python environment is available.
Did you forget to set the RETICULATE_PYTHON environment variable in your .Rprofile before publishing?

My .Rprofile looks exactly like the posted one:
if (Sys.info()[['user']] == 'shiny'){

  # Running on shinyapps.io
  Sys.setenv(PYTHON_PATH = 'python3')
  Sys.setenv(VIRTUALENV_NAME = VIRTUALENV_NAME)
  Sys.setenv(RETICULATE_PYTHON = '/home/shiny/.virtualenvs/forecaster/bin/python')

}

and I am also creating the specific virtual environment in my app.R file. The packages I am trying to include in this environment are: (they should work for python_version <- "3.7.1")
PYTHON_DEPENDENCIES <-  c(
    "pip",
    "mxnet~=1.7",
    "gluonts==0.8.0",
    "numpy",
    "pandas==1.0.5",
    "pathlib==1.0.1",
    "ujson==4.0.2",
    "brotlipy"
)

Therefore, my app.R code looks like this:
virtualenv_dir = Sys.getenv('VIRTUALENV_NAME')
python_path = Sys.getenv('PYTHON_PATH')
reticulate::virtualenv_create(envname = virtualenv_dir,
                                  python  = python_path)
reticulate::virtualenv_install(virtualenv_dir,
                                   packages         = PYTHON_DEPENDENCIES,
                                   ignore_installed = TRUE)
reticulate::use_virtualenv(virtualenv_dir,
                               required = TRUE)

I have tried many methods to work around this issue and none have resulted in my app Python code working in shinyapps.io.
Any help with this will be greatly appreciated!

I'm not sure what is causing the issue. But, have you tried doing print(Sys.getenv(RETICULATE_PYTHON)) in the .Rprofile file, and checking the log on Shinyapps.io to see if you get '/home/shiny/.virtualenvs/forecaster/bin/python'

Yes, this is the log:
2022-03-11T10:56:32.784376+00:00 shinyapps[5812877]: [1] ""
2022-03-11T10:56:32.784456+00:00 shinyapps[5812877]: [1] "/home/shiny/.virtualenvs/forecaster/bin/python"
2022-03-11T10:56:32.919294+00:00 shinyapps[5812877]: Running on host: 85eb2e9541e0
2022-03-11T10:56:32.955869+00:00 shinyapps[5812877]: Server version: 2022.02.0
2022-03-11T10:56:32.955928+00:00 shinyapps[5812877]: shiny version: 1.6.0
2022-03-11T10:56:32.955884+00:00 shinyapps[5812877]: LANG: C.UTF-8
2022-03-11T10:56:32.955911+00:00 shinyapps[5812877]: R version: 4.1.1
2022-03-11T10:56:32.955941+00:00 shinyapps[5812877]: httpuv version: 1.6.2
2022-03-11T10:56:32.955977+00:00 shinyapps[5812877]: rmarkdown version: 2.10
2022-03-11T10:56:32.955990+00:00 shinyapps[5812877]: jsonlite version: 1.7.2
2022-03-11T10:56:32.955998+00:00 shinyapps[5812877]: RJSONIO version: (none)
2022-03-11T10:56:32.955989+00:00 shinyapps[5812877]: knitr version: 1.33
2022-03-11T10:56:32.956006+00:00 shinyapps[5812877]: htmltools version: 0.5.2
2022-03-11T10:56:32.956025+00:00 shinyapps[5812877]: reticulate version: 1.20
2022-03-11T10:56:32.956135+00:00 shinyapps[5812877]: Did you forget to set the RETICULATE_PYTHON environment variable in your .Rprofile before publishing?
2022-03-11T10:56:32.956120+00:00 shinyapps[5812877]: Warning: The reticulate package is installed but no Python environment is available.

I cannot seem to find any solution.

The first line of the log seems to be another print(), but it's blank. Did you have another print() in the Rprofile file? If not, I may suspect that the file is called twice somehow. (but then, I'm still not sure whether that is the cause of this issue. 😓 )
Another thing to double-check is the name of the virtual environment. I see that you were specifying the VIRTUALENV_NAME in a string. I'd double-check that the VIRTUALENV_NAME is set to "forecaster"
VIRTUALENV_NAME = 'forecaster'

....

if (Sys.info()[['user']] == 'shiny'){

  # Running on shinyapps.io
  Sys.setenv(PYTHON_PATH = 'python3')
  Sys.setenv(VIRTUALENV_NAME = VIRTUALENV_NAME)
  Sys.setenv(RETICULATE_PYTHON = paste0('/home/shiny/.virtualenvs/', VIRTUALENV_NAME, '/bin/python'))

}

If you can put your app to a public repo, I'm happy to clone it and take a look. 👍

@nsunami I just made you a collaborator. All the different tests I have tried for uploading to shinyapps.io are in the branch envsetup. Thanks for the help so far

I just saw that you made the repo public! Shall we continue tracking the issue there? I jumped the gun and made an issue here

Still not solved...
The RETICULATE_PYTHON is not recognised on shinyapps.io ... Is there a workaround on that issue because I am starting to become crazy
Thanks in advance,
Best regards,

Same issue.  This is bizarre. How can the same code be running in the demo but not be reproducible?
Is the demo running on a free account? Maybe Posit do not include Python on the free-tier images.
I also tried installing python from within the shiny app and it jut runs out of memory.
It's very annoying that Posit do not have official docs for this. I certainly can't see myself paying for a higher tier at this rate.


Slow loading of HR recruitment page on the Portal

Describe the task
The HR recruitment page on the portal is experiencing significant performance issues, with page load times taking 10+ seconds. This slow loading makes it difficult and frustrating to use the page effectively.
Steps to Reproduce:

Access the HR recruitment page on the portal.

Observe the significant delay in page load times.

Expected behaviour
The HR recruitment page should load within a reasonable timeframe, preferably under a few seconds, to ensure smooth user interaction.
References

Solution to solve this problem:-
As far as I can check there is a lot of queries are running when the HR recruitment page is open. In the hr_application table, if there is a 150 - 200 rows in that table so the minimum time of the page is 1 - 2 seconds.
And If there is 10,000 + data in that table so the minimum time of the page is open is 14 sec. So the total queries 11101 queries are executed. I think there are also unnecessary queries we can remove the ones we don't need. So we need to optimize the index function of the ApplicationController.

@Ayush8923 as per the testing done is the portal it is tasking around 3-4 seconds to load the HR recruitment page.  If the expected time for loading HR recruitment page then you can deploy the feature on the production

@Ayush8923 as per the testing done is the portal it is tasking around 3-4 seconds to load the HR recruitment page. If the expected time for loading HR recruitment page will be between 3-4 seconds, then you can deploy the feature on the production

@deepankar-sajwan Yes. Now it is taking 2 -3 seconds which is correct as compared to earlier. We will do more code cleanup and optimization in it, so I think it will be better.

Reference

@Ayush8923 you can proceed with the code deployment


Upgrade to ONNX 1.15.0

I finally was able to get us to upgrade to ONNX 1.15.0!! YES!! After hours of testing and debugging I found out the problem. Please read below:

I was receiving this error only for RHEL builds of ONNX-MLIR:
/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/internal/api_implementation.py:109: UserWarning: Selected implementation cpp is not available.
  warnings.warn(
Traceback (most recent call last):
  File "/workdir/onnx-mlir/build/docs/doc_example/gen_add_onnx.py", line 1, in <module>
    import onnx
  File "/usr/local/lib64/python3.9/site-packages/onnx/__init__.py", line 75, in <module>
    from onnx import serialization
  File "/usr/local/lib64/python3.9/site-packages/onnx/serialization.py", line 16, in <module>
    import google.protobuf.json_format
  File "/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/json_format.py", line 54, in <module>
    from google.protobuf.internal import type_checkers
  File "/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/internal/type_checkers.py", line 51, in <module>
    from google.protobuf.internal import decoder
  File "/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/internal/decoder.py", line 87, in <module>
    from google.protobuf.internal import encoder
  File "/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/internal/encoder.py", line 71, in <module>
    from google.protobuf.internal import wire_format
  File "/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/internal/wire_format.py", line 36, in <module>
    from google.protobuf import descriptor
  File "/usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/descriptor.py", line 51, in <module>
    from google.protobuf.pyext import _message
ImportError: libprotobuf.so.32: cannot open shared object file: No such file or directory
make[2]: *** [docs/doc_example/CMakeFiles/OMRuntimeTestModel.dir/build.make:73: docs/doc_example/libadd.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:11974: docs/doc_example/CMakeFiles/OMRuntimeTestModel.dir/all] Error 2

The main error is found here:  /usr/lib64/python3.9/site-packages/protobuf-4.21.12-py3.9-linux-s390x.egg/google/protobuf/internal/api_implementation.py:109: UserWarning: Selected implementation cpp is not available.
After searching this warning which is located in google/protobuf/internal/api_implementation.py. There is a block of code for cpp implementation that we are hitting due to the flag but it is failing every time.
if _implementation_type == 'cpp':
  try:
    # pylint: disable=g-import-not-at-top
    from google.protobuf.pyext import _message
    sys.modules['google3.net.proto2.python.internal.cpp._message'] = _message
    _c_module = _message
    del _message
  except ImportError:
    # TODO: fail back to python
    warnings.warn(
        'Selected implementation cpp is not available.')
    pass

It seems like building Protobuf from source has known issues of confusing pure python implementation with cpp implementation hence the error. So when removing the cpp flag we were able to get a successful build. This flag is NOT needed for us to build Protobuf, it is supposed to provide performance enhancement for Protobuf messages but it does not negatively impact us if we remove it.
Here are further links which conclude the issue with using --cpp_implementation:
https://github.com/protocolbuffers/protobuf/issues/9014
https://github.com/protocolbuffers/protobuf/issues/539
https://chromium.googlesource.com/external/github.com/google/protobuf/+/v3.0.0-beta-2/python/

@gongsu832 @AlexandreEichenberger If you have any questions please feel free to let me know but I spent hours debugging for the build to finally go through successfully. Thanks!

No we cannot remove --cpp_implementation. It's there for a reason. It performs much better than the pure python implementation (see https://daobook.github.io/protobuf/docs/performance.html). Onnx uses protobuf extensively (@AlexandreEichenberger @chentong319 @tungld probably know better) for model data so its performance very much impacts what onnx and therefore onnx-mlir does.
It's extremely unlikely that a minor version upgrade for one package (onnx) would require changing the implementation of another third party dependent package (protobuf). The error you get is almost certainly due to factor(s) not related to upgrading onnx itself. If you have trouble making it work, I can try to fix it with another PR.

@gongsu832 According to our git history we did not always have -cpp_implementation so no Gong Su it is NOT REQUIRED. I am not understanding where you are getting that this must be required. Second, I have spent hours working on this. So if you can come up with a better solution, please do. However, I made it clear in the manner which you speak to me needs to have respect. So let's try rephrasing what you sent me, shall we?

You added -cpp-implementation on October 22, 2022 which is last year https://github.com/onnx/onnx-mlir/pull/1752. So then how did things work beforehand?

@tungld @AlexandreEichenberger @chentong319 If you have strong feelings for keeping this flag, please let me know works best for the community. Thanks!

I didn't say --cpp_implementation is required for protobuf to work. I said we cannot remove it because the performance of the pure python implementation after removing --cpp_implementation is abysmal and I have given the link to the performance comparison.
We added the --cpp_implementation flag last year after we found out it's available.
And yes, I will have a better solution that will upgrade onnx without requiring protobuf implementation change.

I didn't say --cpp_implementation is required for protobuf to work. I said we cannot remove it because the performance of the pure python implementation after removing --cpp_implementation is abysmal and I have given the link to the performance comparison.
We added the --cpp_implementation flag last year after we found out it's available.
And yes, I will have a better solution that will upgrade onnx without requiring protobuf implementation change.

Of course you will!

No idea about the impact of --cpp-implementation. Just looked into the setup.py. Could this comment in the code give us any hint?
  if GetOptionFromArgv('--cpp_implementation'):
    # Link libprotobuf.a and libprotobuf-lite.a statically with the
    # extension. Note that those libraries have to be compiled with
    # -fPIC for this to work.

No idea about the impact of --cpp-implementation. Just looked into the setup.py. Could this comment in the code give us any hint?
  if GetOptionFromArgv('--cpp_implementation'):
    # Link libprotobuf.a and libprotobuf-lite.a statically with the
    # extension. Note that those libraries have to be compiled with
    # -fPIC for this to work.

Yeah libtools should take care of that for us but the above error also mentions:
ImportError: libprotobuf.so.32: cannot open shared object file: No such file or directory
When I do a find / -name "libprotobuf.so.32" -print 2> /dev/null \ of libprotobuf.so.32 the file does exists and I also did a check to see if there were conflicting protobuf versions being pulled in by ONNX and now that we are on the same version as them... there are no conflicting versions installed. Also ONNX only installs protobuf if they cannot detect that a version is already installed.
/usr/local/lib/libprotobuf.so.32
/workdir/protobuf/src/.libs/libprotobuf.so.32
-rwxr-xr-x 1 root root     1012 Nov 17 11:23 /usr/local/lib/libprotobuf-lite.la
lrwxrwxrwx 1 root root       27 Nov 17 11:23 /usr/local/lib/libprotobuf-lite.so -> libprotobuf-lite.so.32.0.12
lrwxrwxrwx 1 root root       27 Nov 17 11:23 /usr/local/lib/libprotobuf-lite.so.32 -> libprotobuf-lite.so.32.0.12
-rwxr-xr-x 1 root root  6331648 Nov 17 11:23 /usr/local/lib/libprotobuf-lite.so.32.0.12
-rwxr-xr-x 1 root root      982 Nov 17 11:23 /usr/local/lib/libprotobuf.la
lrwxrwxrwx 1 root root       22 Nov 17 11:23 /usr/local/lib/libprotobuf.so -> libprotobuf.so.32.0.12
lrwxrwxrwx 1 root root       22 Nov 17 11:23 /usr/local/lib/libprotobuf.so.32 -> libprotobuf.so.32.0.12
-rwxr-xr-x 1 root root 40894872 Nov 17 11:23 /usr/local/lib/libprotobuf.so.32.0.12
-rwxr-xr-x 1 root root     1000 Nov 17 11:23 /usr/local/lib/libprotoc.la
lrwxrwxrwx 1 root root       20 Nov 17 11:23 /usr/local/lib/libprotoc.so -> libprotoc.so.32.0.12
lrwxrwxrwx 1 root root       20 Nov 17 11:23 /usr/local/lib/libprotoc.so.32 -> libprotoc.so.32.0.12
-rwxr-xr-x 1 root root 56825384 Nov 17 11:23 /usr/local/lib/libprotoc.so.32.0.12


Patch seems to break (or is ignored by) testcoverage tooling?

First off, I really liked the API/implementation of Patch. Really enjoy using it so thanks for making it
But whilst experimenting with some code coverage tooling (mix test --cover) noticed that code that uses Patch.patch disappears from the reports.
So decided to clone this repo and noticed the same. Picked a test at random that uses patch(), refute_any_call_test.exs
❯ grep 'patch(' ./test/user/refute_any_call_test.exs
      patch(RefuteAnyCall, :function_with_multiple_arities, :patched)
      (..)
      patch(RefuteAnyCall, :other_function, :patched)
      patch(RefuteAnyCall, :function_with_multiple_arities, :patched)

but then:
mix test --cover 2>&1 | grep RefuteAny

outputs nothing, same for
grep RefuteAnyCall cover -r

which, seeing all the generated html files in cover/ seems unlikely
I assume this is known behaviour? I can imagine its a side-effect of how the 'super-powers' work but are you aware of a workaround or fix? Because its a bit ironic that the most well tested code in my codebase shows up as 0% or not at all :)
And a ps. Running the test directly mix test --cover ./test/user/refute_any_call_test.exs does show the User.Patch modules nearby but almost all of them as 0% coverage

thanks @ihumanable , but you think its fixable? That's good to hear then, thanks

I think so, I've actually got some time today to take a look so 🤞


Typo in podman-build man page

/kind bug
Description
Here https://github.com/containers/libpod/blob/master/docs/podman-build.1.md and in the podman-build man page, --ulimit usage is this

--ulimit=type:soft-limit[:hard-limit]

while in the subsequent example, and actually using the option in the command line, the right syntax should be

--ulimit=type=soft-limit[:hard-limit]

Closed via #1327


Track multiple edition of a Pud

If a PUD open in one editor has been modified from outside, the editor should been about the file being modified by somone else.

That's not easy to do. There is the Ecore_File_Monitor, but it does not differenciate the origins of the modifications. It is not a fantastic improvement, so don't make it a showstopper.

inotify or open an exclusive lock on the file so no others can write to it

Thanks for these ideas, unfortnuately Inotify is not cross-plaform , and I want war2edit to run on macOS.
I haven't considered the exclusive lock, maybe that's the way to go, but then again there is the cross-platform consideration. C11 introduced the "x" parameter to fopen(), but this would require C11 compilers and libc.


CSG build pipeline

Spawn a top-level CSG combiner node, then populate it with CSGMesh nodes in Union mode using brush vertex data.
This should cull away any unnecessary internal faces, but might be a bit performance-heavy at build time and subject to edge cases.

This isn't feasible. I did some experimenting by swapping out the MeshInstance nodes from the new per-brush pipeline with CSGMesh nodes, then manually grouping them under a CSGCombiner.
The resulting mesh was completely unusable- holes everywhere, and with lacking internal face culling to boot. Closing.


[Feature Request]: Support multiple fields in belongsTo, hasMany and hasOne

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v16.14.0
Amplify CLI Version
7.6.25
What operating system are you using?
Windows
Amplify Codegen Command
codegen models
Describe the bug
Having multiple fields results in the error DataStore only support one key in field. Using belongsTo and hasOne with composite keys is therefore not possible.
Expected behavior
Generate the models with composite keys.
Reproduction steps
This is a new feature request.
GraphQL schema(s)

# Put schemas below this line
type Post @model {
  tenant: ID! @primaryKey(sortKeyFields: ["id"])
  id: ID!
  title: String!
  comments: [Comment] @hasMany(indexName: "byPost", fields: ["tenant", "id"])
}

type Comment @model {
  tenant: ID!
    @primaryKey(sortKeyFields: ["id"])
    @index(name: "byPost", sortKeyFields: ["postID"])
  id: ID!
  postID: ID!
  content: String!
  post: Post @belongsTo(fields: ["tenant", "postID"])
}

Log output

# Put your logs below this line

Additional information
No response

Hi @MatthiasF999, could you share a minimum reproduction schema?

My mistake, I missed the expander. Thanks

I also encountered this error.
But I perform data validation and query.
This exception is thrown when we execute the "amplify push" command to push and does not affect the query, you can still query.
E.g:
Post has multiple comments
comment has corresponding Post
But for fear of other influences or instability, I still removed the way @belongsTo uses multiple fields.

This feature will be soon supported by the ongoing custom primary key feature.

Using multiple belongsTo in one model, I tried to change to add multiple belongsTo on variables but, somehow multiple belongsTo broke toJson in related model. If you fix toJson, I can add my pull request for this

Do you know exactly when this feature will be supported ? I really need this fix to be honest.

I created new GraphQLRequest and add my modified and missing id to it then using this for mutation. I just lookup belongsTo and other ID's variable creation on the amolify_api and i modified there but i couldn't handle json decode, don't know how I couldn't parse other id.it's there but i cannot get the id. İt's interesting. Anyway i temporarily fix by adding missing id by hand. If you think about to fix this issue like this, i can add example for you


Inconsistency between EasyMarkdownEditor's preview and actual effect

I am an organizer, here is the our competition link: https://www.codabench.org/competitions/edit/3283
In the "Pages" editor, EasyMarkdownEditor did not render the HTML with style, but in the final published page, it will.
I put a PoC here:
<div style="width:200px; height: 200px; background: red">
</div>

This is a non-urgent issue that just causes confusions for the organizers.

In front of the page there is an eye icons, when you press that button, you can see how the page will look like without saving changes. Markdown editor is not showing the style because it is built to render only Markdown.

I am closing this because we have many bugs to fix and features to implement. If you want to contribute and fix this, we would appreciate your contribution. We may want to check the markdown thing in detail in the next big update of the UI.