Fix pep8/pylint prospector errors

This series of commits fixes a handful of pylint/pep8 errors discovered by the latest version of prospector (1.2.0).
After running prospector in Tern's root directory with these changes, there are no longer an prospector error messages.
Fixes #513
Signed-off-by: Rose Judge<EMAIL_ADDRESS>

@rnjudge Commit titles for 1, 2 and 5 sound similar. Any way you can indicate what pep8 error they are fixing?

@rnjudge Commit titles for 1, 2 and 5 sound similar. Any way you can indicate what pep8 error they are fixing?

Sure

@rnjudge Commit titles for 1, 2 and 5 sound similar. Any way you can indicate what pep8 error they are fixing?

Sure

@nishakm Done :)

@rnjudge looks good! I'll merge this via a merge commit.


only raise piggyback count if ping is succesfull

Not ready for review yet, only want to see if it passes the tests

ready for review @uber/ringpop


export all typescipt types

Could you please consider exporting all of your typescript types and interfaces?

Hey @Bearsie, could you please clarify what you meant by exporting the Typescript types and interfaces? There is an index.d.ts file included in the project.

Thanks for clarifying. My understanding was that index.d.ts was supposed to describe only the public exports of the module. But I can see a use case for wanting to reuse some of the types in your own code.


Multitenant message indicator shows read message as unread in other tenant

Hi I am working for two companies, we use both Teams.
I am logged into company A
When I get a message in the other company B.
A small one in a red circle appears at the name of the other company on the upper right area of teams.
Great!
I go there (Company B) and read the message, then I go back to the other company. (Company A)
But the message indicator still shows this message. Even if it is already read.
Please fix. Thx

@TeamStriebigSebastian Thanks for reporting this. We are looking into it.

@TeamStriebigSebastian This is a known limitation. Cross-tenant pill count is currently updated every 15 minutes. We are working on improving this experience. Could you please check if it updates within 15 mins or takes more than that?

Lol, it pops up when I log in in the morning and switch to a guest tenant.
Then it shows me that my home tenant has a message for me, I switch back no
message.
So no the 15 minutes thing is not working.
Better built a non-time bound process in a real-time messaging system...
On Wed, 21 Aug 2019 at 06:05, Gousia-MSFT<EMAIL_ADDRESS>wrote:

@TeamStriebigSebastian https://github.com/TeamStriebigSebastian This is
a known limitation. Cross-tenant pill count is currently updated every 15
minutes. We are working on improving this experience. Could you please
check if it updates within 15 mins or takes more than that?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/msteams-docs/issues/853?email_source=notifications&email_token=AGMSZMSNZZENN3MRAL7L3DTQFS5J3A5CNFSM4IINXPJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4YLRGA#issuecomment-523286680,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGMSZMRBCAJ2SXI4W7V53QTQFS5J3ANCNFSM4IINXPJQ
.

This is an unfortunate design gap on how the service works today. We're planning to sunset this service (for this and a variety of other reasons) and move to a new service but we do not have a firm timeline to share.


Unifying shapes of tensors throughout the library

Rethink the diff function
At the core of the neurodiffeq library is the diff(x, t) function, which computes the partial derivative ∂x/∂t evaluated at t. Usually, both tensor t and x have shapes of (n_samples, 1). When either x.shape or t.shape is malformed, however, there are cases where things could go wrong due to broadcasting. Such cases are so subtle that they have gone unnoticed for a long time.
All our generators (as defined in neurodiffeq.generator) currently return tensors with shapes (n_samples,) instead of (n_samples, 1). Efforts should be put into unifying the tensor shapes everywhere.
Here are two simple cases for review.
Case 1: Shapes don't matter
In this case, we try different combinations of x.shape and t.shape and check the shape of the output ∂x/∂t, namely:

[n, 1] and [n] --> [n]
[n]    and [n]--> [n]
[n, 1] and [n, 1]--> [n,1]
[n]    and [n, 1]--> [n,1]

To see this, run the following code. Note that d1, d2, d3, and d4, while having different shapes, hold the same values. This is the reason why we incorrectly believed in the soundness of the diff() function.
n = 10

t = torch.rand(n, requires_grad=True)
x = torch.sin(t)
d1 = diff(x.reshape(-1, 1), t)
d2 = diff(x.reshape(-1), t)

t = t.reshape(-1, 1)
x = torch.sin(t)
d3 = diff(x.reshape(-1, 1), t)
d4 = diff(x.reshape(-1), t)

Case 2: Shapes matter
In this second case, we examine two new operators – div and curl in spherical coordinates – and show that only when x.shape and t.shape are both (n, 1) will the vector identity div(curl(...)) == 0 hold.
Here is the definition of curl and divergence in spherical coordinates
# these two operators have been recently implemented in neurodiffeq.operators
def spherical_curl(u_r, u_theta, u_phi, r, theta, phi):
    d_r = lambda u: diff(u, r)
    d_theta = lambda u: diff(u, theta)
    d_phi = lambda u: diff(u, phi)

    curl_r = (d_theta(u_phi * sin(theta)) - d_phi(u_theta)) / (r * sin(theta))
    curl_theta = (d_phi(u_r) / sin(theta) - d_r(u_phi * r)) / r
    curl_phi = (d_r(u_theta * r) - d_theta(u_r)) / r

    return curl_r, curl_theta, curl_phi

def spherical_div(u_r, u_theta, u_phi, r, theta, phi):
    div_r = diff(u_r * r ** 2, r) / r ** 2
    div_theta = diff(u_theta * sin(theta), theta) / (r * sin(theta))
    div_phi = diff(u_phi, phi) / (r * sin(theta))
    return div_r + div_theta + div_phi

Here we define a vector field q by specifying the rule to compute q given coordinates (r, theta, phi)
def compute_q(r, theta, phi):
    r_theta_phi = torch.stack([r.flatten(), theta.flatten(), phi.flatten()], dim=1)
    W = torch.tensor([
        [.01, .04, .07],
        [.02, .05, .08],
        [.03, .06, .09],
    ])
    q = torch.matmul(r_theta_phi, W)
    q = torch.tanh(q)
    return q[:, 0], q[:, 1], q[:, 2]

We then test the vector identity div(curl(q)) == 0 for q
n = 10

# create r, theta, and phi with shape (n, 1)
r = torch.rand(n, 1, requires_grad=True) + 0.1
theta = torch.rand(n, 1, requires_grad=True) * np.pi
phi = torch.rand(n, 1, requires_grad=True)  * np.pi * 2
q_r, q_theta, q_phi = compute_q(r, theta, phi)

# bind the operators to the r, theta, phi created above
div = lambda u_r, u_theta, u_phi: spherical_div(u_r, u_theta, u_phi, r, theta, phi)
curl = lambda u_r, u_theta, u_phi: spherical_curl(u_r, u_theta, u_phi, r, theta, phi)

div_curl_q1 = div(*curl(q_r.reshape(-1, 1), q_theta.reshape(-1, 1), q_phi.reshape(-1, 1)))
div_curl_q2 = div(*curl(q_r.reshape(-1), q_theta.reshape(-1), q_phi.reshape(-1)))

# create r, theta, and phi with shape (n,)
r = r.reshape(-1)
theta = r.reshape(-1)
phi = r.reshape(-1)
q_r, q_theta, q_phi = compute_q(r, theta, phi)

# bind the operators to the r, theta, phi created above
div = lambda u_r, u_theta, u_phi: spherical_div(u_r, u_theta, u_phi, r, theta, phi)
curl = lambda u_r, u_theta, u_phi: spherical_curl(u_r, u_theta, u_phi, r, theta, phi)

div_curl_q3 = div(*curl(q_r.reshape(-1, 1), q_theta.reshape(-1, 1), q_phi.reshape(-1, 1)))
div_curl_q4 = div(*curl(q_r.reshape(-1), q_theta.reshape(-1), q_phi.reshape(-1)))

print(div_curl_q1, div_curl_q2, div_curl_q3, div_curl_q4, sep="\n")

Printing all four div_curl_qs will show that, only div_curl_q1 is (approximately) equal to 0, which means both the dependent and independent variables must have shape (n, 1) for the differentiation to go correctly.

After discussing with David, we decide to change the default behavior of the diff function.

Before v0.2.0, we'll be using these (python) functions for differentiation:

safe_diff: performs shape checking
unsafe_diff: doesn't perform shape checking; behaves exactly like the original diff
diff: defaults to unsafe_diff; but issues a one-time warning that, in the future, it will default to safe_diff

Starting from v0.2.0

safe_diff: performs shape checking
unsafe_diff: doesn't perform shape checking; behaves exactly like the original diff
diff: defaults to safe_diff, no warning will be issued


Add logic for taxes - totals, prices

It requires probably diging into mage2nosql tax synchronization logic

Started investigation for that

https://magento.stackexchange.com/questions/198378/magento-2-rest-api-get-taxrate-by-tax-class-id
http://devdocs.magento.com/swagger/index.html#!/taxTaxRuleRepositoryV1
It's getting quite complicated!

We need separate class TaxCalculator or so on - doing the Magento's logic and which can be mixed in to PoJ data objects asa a mixin to have "priceInclTax()" getter or sth like this

Or just vue mixin that'll use this calculator?

Tax sync added: https://github.com/DivanteLtd/mage2nosql/commit/d434d6921c529f86123f04e16c3c819f195def7d

https://github.com/DivanteLtd/vue-storefront/pull/294


Scheduler does not appear after installing

Description

After installing the actual version of jupyter_scheduler the Sceduler button does not appear:

The installation of jupyter-scheduler went through without errors.
Reproduce

Install 'jupyter_scheduler' in Jupyter Lab
Restart the kernel.
Stop the Jupyter Lab
Start the Jupyter Lab
Open the Jupyter lab
Use the sceduler button --> not button appears

Expected behavior

The button should appear like this:

Context

Operating System and version:

Browser and version: Google Chrome Version 113.0.5672.127

Jupyter Server version: Version 3.4.8

Troubleshoot Output

Paste the output from running `jupyter troubleshoot` from the command line here.
You may want to sanitize the paths in the output.

Command Line Output

Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.

Browser Output

Paste the output from your browser Javascript console here, if applicable.

@njuhl76 Thanks for opening this issue! How did you install jupyter_scheduler? We recommend pip install jupyter_scheduler.
What do you see when you run the following two commands?

jupyter server extension list
jupyter labextension list

thank you for your answer!
Here are my results:
(base) jupyter@p-beat-ai--nils-juhl--notebook:~$ jupyter server extension list
Config dir: /home/jupyter/.jupyter
Config dir: /opt/conda/etc/jupyter
jupyter_scheduler enabled
- Validating jupyter_scheduler...
jupyter_scheduler 1.3.2 OK
jupyter_server_mathjax enabled
- Validating jupyter_server_mathjax...
jupyter_server_mathjax  OK
nbclassic enabled
- Validating nbclassic...
nbclassic 0.5.3 OK
nbdime enabled
- Validating nbdime...
nbdime 3.1.1 OK
notebook_shim enabled
- Validating notebook_shim...
notebook_shim  OK
Config dir: /usr/local/etc/jupyter

(base) jupyter@p-beat-ai--nils-juhl--notebook:~$ jupyter labextension list
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir] [--paths] [--json] [--debug] [subcommand]
Jupyter: Interactive Computing
positional arguments:
subcommand     the subcommand to launch
optional arguments:
-h, --help     show this help message and exit
--version      show the versions of core jupyter packages and exit
--config-dir   show Jupyter config dir
--data-dir     show Jupyter data dir
--runtime-dir  show Jupyter runtime dir
--paths        show all Jupyter paths. Add --json for machine-readable format.
--json         output paths as machine-readable json
--debug        output debug information about paths
Available subcommands: bundlerextension dejavu execute kernel kernelspec migrate nbclassic nbconvert nbextension notebook run server serverextension troubleshoot trust
Jupyter command jupyter-labextension not found.

Here is my Result:

Per your results, it looks like JupyterLab is not installed. Are you sure you're running JupyterLab, or are you using Jupyter Notebook instead? The scheduler requires JupyterLab, not Notebook or other apps.

Thanks for the answer!
Now I have installed jupyterlab:

After that I did that again and it did not work again:

Install 'jupyter_scheduler' in Jupyter Lab
Restart the kernel.
Stop the Jupyter Lab
Start the Jupyter Lab
Open the Jupyter lab
Use the sceduler button --> not button appears

Do you have any idea, how to deal with these errors @JasonWeill ?

@njuhl76 I don't see any actionable errors in the log above, and I have not tried to use Jupyter Scheduler on Google Compute Engine. (I also work for AWS, a competitor of Google Cloud.)

@njuhl76 Hi Nils, we don't have access to GCP, so unfortunately, we're not able to reproduce your issue. From the data you've posted, it is apparent that the Jupyter Scheduler extension is being successfully initialized; this can be seen from the log line:
[I 2023-06-29 16:39:16.480 ServerApp] jupyter_scheduler | extension was successfully loaded.

However, it also looks like you are spawning two JupyterLab servers. This is apparent because port 8080 is already in use:
[I 2023-06-29 16:39:16.783 ServerApp] The port 8080 is already in use, trying another port.
[I 2023-06-29 16:39:16.784 ServerApp] Serving notebooks from local directory: /home/jupyter
[I 2023-06-29 16:39:16.784 ServerApp] Jupyter Server 1.23.6 is running at:
[I 2023-06-29 16:39:16.784 ServerApp] http://localhost:8081/lab
[I 2023-06-29 16:39:16.784 ServerApp]  or http://<IP_ADDRESS>:8081/lab
[I 2023-06-29 16:39:16.784 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 2023-06-29 16:39:16.790 ServerApp] No web browser found: could not locate runnable browser.`

My guess is that you are actually spawning two JupyterLab servers, one at :8080 (the "main" JL instance) and one at :8081. You are likely connecting to the one at :8080, and thus aren't seeing the extension that was installed on the instance at :8081. You will want to find the Conda environment that the main JL instance is hosted in, install the package there, and then restart the main JL instance.
To list available Conda environments:
conda env list

To switch to a new Conda environment:
conda activate <env-name>

It looks like GCP Deep Learning VM has all Jupyter-related packages in a separate Conda environment. See here:

Reference: https://cloud.google.com/deep-learning-vm/docs/release-notes
Since this issue appears to be specifically related to GCP's Conda environments and not Jupyter Scheduler itself, I will close the issue.

Thank you for these insightful comments @dlqqq !
In fact, I had the M105 version running. On this then also two environment existed. In the Jupyter environment I can run the sceduler.
Thanks again @dlqqq  & @JasonWeill


This hopefully partially fixes the #15003
The other part - disabling "rolling builds" has to be done in AppVeyor UI.

Hmm, apparently this does not work as expected :(


have a question

I trained a vit-b-32 and a vit-b-32-soft-moe with 64 expert on our own training set, with the same configuration, but I feel that the accuracy of the two is similar.

My training set has about 10 million data

@daixiangzi would you like to share your training curve?

@daixiangzi would you like to share your training curve?

I didn’t draw the loss curve, but I did compare the loss logs. They are almost the same. I can provide some of the moe codes that I modified.

class QuickGELU(nn.Module):
    def forward(self, x: torch.Tensor):
        return x * torch.sigmoid(1.702 * x)

def FeedForward(dim):
        dim_hidden = int(dim * 4)
        return nn.Sequential(nn.Linear(dim, dim_hidden),QuickGELU(),nn.Dropout(p=0.2),nn.Linear(dim_hidden, dim),nn.Dropout(p=0.2))
class MultiExpertLayer_v2(nn.Module):
    def __init__(
        self,
        in_features: int,
        out_features: int,
        num_experts: int,
        bias: bool = True,
        device: Optional[Union[torch.device, str]] = None,
        dtype: Optional[torch.dtype] = None,
    ):
        super().__init__()
        self.in_features = in_features
        self.experts = torch.nn.ModuleList([FeedForward(in_features).to(device) for i in range(num_experts)])

    def forward(self,x):
        #x:10, 128, 1, 768
        out = [fn(x[:,i,:,:]) for i,fn in enumerate(self.experts)]
        #[128, b, 1, d])
        out = torch.stack(out)
        out = einsum(out ,"e b s d-> b e s d")
        #[b, 128, 1, d])
        return out

class SoftMoE(nn.Module):
    """A PyTorch module for Soft-MoE, as described in the paper:
        "From Sparse to Soft Mixtures of Experts"
        https://arxiv.org/pdf/2308.00951.pdf

    einstein notation:
    - b: batch size
    - m: input sequence length
    - d: embedding dimension
    - n: num experts
    - p: num slots per expert
    - (n * p): total number of slots

    Args:
        embed_dim (int): embedding dimension (d)
        num_experts (int): number of experts (n)
        slots_per_expert (int): number of slots per expert (p)
        bias (bool): whether to include a bias term. Default: True.
    """

    def __init__(
        self,
        in_features: int,
        out_features: int,
        num_experts: int,
        slots_per_expert: int,
        bias: bool = True,
        device: Optional[Union[str, torch.device]] = None,
        dtype: Optional[torch.dtype] = None,
    ):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.num_experts = num_experts
        self.slots_per_expert = slots_per_expert
        self.bias = bias
        #387,128,1
        self.phi = nn.Parameter(
            torch.empty(
                (in_features, num_experts, slots_per_expert),
                device=device,
                dtype=dtype,
            )
        )
        self.scale = nn.Parameter(torch.ones(1))
        self.experts = MultiExpertLayer_v2(
            in_features=in_features,
            out_features=out_features,
            num_experts=num_experts,
            bias=bias,
            device=device,
            dtype=dtype,
        )

        self.reset_parameters()

    def reset_parameters(self) -> None:
        # NOTE: Copy weight initialization from 'nn.Linear.reset_parameters'
        # TODO: Check for initialization strategy from the paper
        nn.init.kaiming_uniform_(self.phi, a=math.sqrt(5))

    def normalize(self,x:torch.Tensor,axis:int):
        return x*torch.rsqrt(torch.square(x).sum(dim=axis, keepdim=True) + 1e-6)

    def forward(self, x: Tensor) -> Tensor:
        """Forward pass for the Soft-MoE layer, as described in:
            https://arxiv.org/pdf/2308.00951.pdf
        See: equations (1-3), algorithm 1, and figure 2

        einstein notation:
        - b: batch size
        - m: input sequence length
        - d: embedding dimension
        - n: num experts
        - p: num slots per expert
        - (n * p): total number of slots

        Args:
            x (Tensor): input tensor of shape (b, m, d)

        Returns:
            Tensor: output tensor of shape (b, m, d)
        """
        if x.size(-1) != self.in_features:
            raise ValueError(
                f"Expected x.size(-1)={x.size(-1)} to match embed_dim={self.in_features}, "
                f"but got {x.size(-1)}."
            )
        elif x.ndim != 3:
            raise ValueError(f"Expected input to have 3 dimensions, but got {x.ndim}.")
        #logits:[50, 1, 128, 1]
        #norm x and phi
        #x: [b,50,768]
        #phi [768, 128, b]
        phi = self.phi
        phi = self.scale*self.normalize(phi,axis=0)
        x = self.normalize(x,axis=-1)
        #logits:[b, 50, 128, 1]
        logits = einsum(x, phi, "b m d, d n p -> b m n p")
        #add noise
        if self.training:
            normal = torch.randn(logits.shape, dtype=logits.dtype,device=logits.device)
            logits = logits +(1.0 * normal)
        #dispatch_weights:[b, token_num, 128, 1])
        dispatch_weights = logits.softmax(dim=1)  # denoted 'D' in the paper
        # NOTE: The 'torch.softmax' function does not support multiple values for the
        # 'dim' argument (unlike jax), so we are forced to flatten the last two dimensions.
        # Then, we rearrange the Tensor into its original shape.
        #对experts softmax:combine_weights:[b, token_num, 128, 1]
        combine_weights = rearrange(logits.flatten(start_dim=2).softmax(dim=-1),"b m (n p) -> b m n p",n=self.num_experts)

        # NOTE: To save memory, I don't rename the intermediate tensors Y, Ys, Xs.
        # Instead, I just overwrite the 'x' variable.  The names from the paper are
        # included in a comment for each line below.

        x = einsum(x, dispatch_weights, "b m d, b m n p -> b n p d")  # Xs
        #x:[b,128, 1, out_d]
        x = self.experts(x)  # Ys
        #combine_weights:[b, 50, 128, 1]
        x = einsum(x, combine_weights, "b n p d, b m n p -> b m d")  # Y
        #[b, 128, 1, 384]
        return x

    def extra_repr(self) -> str:
        return (
            f"in_features={self.in_features}, out_features={self.out_features}, "
            f"num_experts={self.num_experts}, slots_per_expert={self.slots_per_expert}, "
            f"bias={self.bias}"
        )

you don't track your experiments? you need to provide training and validation curves.


Add icons and support i18n in typedInput of JSON editor

[x] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)

Proposed changes
I found that there're menus that don't support internationalization and have no icons in JSON editor. Therefore, I fixed them.
Checklist

[x] I have read the contribution guidelines
[ ] For non-bugfix PRs, I have discussed this change on the mailing list/slack team.
[x] I have run grunt to verify the unit tests pass
[ ] I have added suitable unit tests to cover the new/changed functionality

Coverage remained the same at 77.586% when pulling<PHONE_NUMBER>925d5f5cfb99483ca0703ad929a8a8 on kazuhitoyokoi:master-fixjsoneditor into c1dae95f71473f75f6d50ff005894e3f084c79f5 on node-red:master.


derive_mirror_rnda errors

derive_mirro_rnda is not working for me, see error below.
There is a fix this in that line in the code, so this is probably known?
python applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075
INFO::db_handler(l579)::_getSiteParametersYaml::Reading DB file /workdir/external/simulation-model-description/configReports/parValues-Sites.yml
Traceback (most recent call last):
  File "applications/derive_mirror_rnda.py", line 227, in <module>
    tel.changeParameter("mirror_list", args.mirror_list)
  File "/workdir/external/gammasim-tools/simtools/model/telescope_model.py", line 415, in changeParameter
    if not isinstance(value, type(self._parameters[parName]["Value"])):
TypeError: string indices must be integers

This happens without using the DB right?

'useMongoDB: true' , so that should not be the issue.
I really struggle setting all the values in config.yml correctly, maybe we can improve the documentation and the error messages.
e.g., instead of writing:
ERROR::config(l84)::get::Config does not contain dataLocation

Write:
ERROR::config(l84)::get::Config keyword dataLocation not found in file <full path to file>

It took me a while to find out that dataLocation is a required keyword (for some reasons I had testdataLocation in my config.yaml file), and then I did not know from which directory the config.yml file was read.
Note that the full error message here is very confusing:
python gammasim-tools/applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075
<EMAIL_ADDRESS>password:
ERROR::config(l84)::get::Config does not contain dataLocation
Traceback (most recent call last):
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 305, in <module>
    meanD80, sigD80 = run(rndaStart)
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 235, in run
    ray = RayTracing.fromKwargs(
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 198, in fromKwargs
    return cls(**args, configData=configData)
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 126, in __init__
    _parameterFile = io.getDataFile("parameters", "ray-tracing_parameters.yml")
  File "/workdir/external/gammasim-tools/simtools/io_handler.py", line 187, in getDataFile
    Path(cfg.get("dataLocation")).joinpath(parentDir).joinpath(fileName).absolute()
  File "/workdir/external/gammasim-tools/simtools/config.py", line 85, in get
    raise KeyError()
KeyError
INFO::db_handler(l187)::_closeSSHTunnel::Closing SSH tunnel(s)

It doesn't find a keyword in a config.yml, but most of the error message is about a database connection and ssh issues.
Don't worry about the error message itself. I think all issues I have is that files are not found in the modelFilesLocations. This is probably an error on how I set it up on my side (and we need to get this stuff into the database).

I think I get it now: dataLocation needs to point to 'gammasim-tools/data' and it contains configuration files required to run gammasim-tools. That was completely unexpected!
As a user, I did not expect that anything else is needed when running e.g.,
python gammasim-tools/applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075

This is quite a rich command line (nothing wrong with that), but it does not propose that a file name 'data/parameters/ray-tracing_parameters.yml' is required to run the application.
Why did it work for me a couple of months ago? I think the reason is that now I don't start the applications from the gammasim-tools directory but from somewhere else (which means the default paths are not set correctly).

I think I get it now: dataLocation needs to point to 'gammasim-tools/data' and it contains configuration files required to run gammasim-tools. That was completely unexpected!

The 'data' directory is need for many things. It is where we store many parameter files, layout files, test files etc.
I made it a config entry because we might want to move it somewhere in the future, but by now we keep it inside the main repo.

As a user, I did not expect that anything else is needed when running e.g.,
python gammasim-tools/applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075

This is quite a rich command line (nothing wrong with that), but it does not propose that a file name 'data/parameters/ray-tracing_parameters.yml' is required to run the application.

It should not. This file is needed anytime one needs to run a ray tracing module.

Why did it work for me a couple of months ago? I think the reason is that now I don't start the applications from the gammasim-tools directory but from somewhere else (which means the default paths are not set correctly).

Yes, that is probably the reason. We could use the full path as the default for the data dir in the config file. I will open an issue about it.

Note that the full error message here is very confusing:
python gammasim-tools/applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075
<EMAIL_ADDRESS>password:
ERROR::config(l84)::get::Config does not contain dataLocation
Traceback (most recent call last):
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 305, in <module>
    meanD80, sigD80 = run(rndaStart)
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 235, in run
    ray = RayTracing.fromKwargs(
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 198, in fromKwargs
    return cls(**args, configData=configData)
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 126, in __init__
    _parameterFile = io.getDataFile("parameters", "ray-tracing_parameters.yml")
  File "/workdir/external/gammasim-tools/simtools/io_handler.py", line 187, in getDataFile
    Path(cfg.get("dataLocation")).joinpath(parentDir).joinpath(fileName).absolute()
  File "/workdir/external/gammasim-tools/simtools/config.py", line 85, in get
    raise KeyError()
KeyError
INFO::db_handler(l187)::_closeSSHTunnel::Closing SSH tunnel(s)

It doesn't find a keyword in a config.yml, but most of the error message is about a database connection and ssh issues.

The error is as clear as we can make it. It is a bit confusing because of python. The large block about KeyError is inevitable, and the one line about the DB is because we close the connection after the error.
What we can do is to make the line "ERROR::config(l84)::get::Config does not contain dataLocation" a bit clearer as you suggested. I will create a issue for that and fix it next.

Is it working in the end, @GernotMaier ?

Is it working in the end, @GernotMaier ?

Yes, but it took me two hours. I think this shows you that the setup is ideal for users who are not very familiar with the system.

Note that the full error message here is very confusing:
python gammasim-tools/applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075
<EMAIL_ADDRESS>password:
ERROR::config(l84)::get::Config does not contain dataLocation
Traceback (most recent call last):
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 305, in <module>
    meanD80, sigD80 = run(rndaStart)
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 235, in run
    ray = RayTracing.fromKwargs(
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 198, in fromKwargs
    return cls(**args, configData=configData)
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 126, in __init__
    _parameterFile = io.getDataFile("parameters", "ray-tracing_parameters.yml")
  File "/workdir/external/gammasim-tools/simtools/io_handler.py", line 187, in getDataFile
    Path(cfg.get("dataLocation")).joinpath(parentDir).joinpath(fileName).absolute()
  File "/workdir/external/gammasim-tools/simtools/config.py", line 85, in get
    raise KeyError()
KeyError
INFO::db_handler(l187)::_closeSSHTunnel::Closing SSH tunnel(s)

It doesn't find a keyword in a config.yml, but most of the error message is about a database connection and ssh issues.

The error is as clear as we can make it. It is a bit confusing because of python. The large block about KeyError is inevitable, and the one line about the DB is because we close the connection after the error. What we can do is to make the line "ERROR::config(l84)::get::Config does not contain dataLocation" a bit clearer as you suggested. I will create a issue for that and fix it next.

Why can't the program exit in a clean way after there is a configuration file has not been found? This would be much clearer.

Note that the full error message here is very confusing:
python gammasim-tools/applications/derive_mirror_rnda.py --site North --telescope MST-FlashCam-D --mean_d80 1.4 --sig_d80 0.16 --mirror_list mirror_MST_focal_lengths.dat --d80_list mirror_MST_D80.dat --rnda 0.0075
<EMAIL_ADDRESS>password:
ERROR::config(l84)::get::Config does not contain dataLocation
Traceback (most recent call last):
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 305, in <module>
    meanD80, sigD80 = run(rndaStart)
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 235, in run
    ray = RayTracing.fromKwargs(
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 198, in fromKwargs
    return cls(**args, configData=configData)
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 126, in __init__
    _parameterFile = io.getDataFile("parameters", "ray-tracing_parameters.yml")
  File "/workdir/external/gammasim-tools/simtools/io_handler.py", line 187, in getDataFile
    Path(cfg.get("dataLocation")).joinpath(parentDir).joinpath(fileName).absolute()
  File "/workdir/external/gammasim-tools/simtools/config.py", line 85, in get
    raise KeyError()
KeyError
INFO::db_handler(l187)::_closeSSHTunnel::Closing SSH tunnel(s)

It doesn't find a keyword in a config.yml, but most of the error message is about a database connection and ssh issues.

The error is as clear as we can make it. It is a bit confusing because of python. The large block about KeyError is inevitable, and the one line about the DB is because we close the connection after the error. What we can do is to make the line "ERROR::config(l84)::get::Config does not contain dataLocation" a bit clearer as you suggested. I will create a issue for that and fix it next.

Why can't the program exit in a clean way after there is a configuration file has not been found? This would be much clearer.

I will try to get rid of this big block about the KeyError, but I'm not sure whether that is possible.
To me this message looks like a pretty common python error. And it is actually more clear than many other packages because we added the customized mesage about the config file, that is not hard to find.
Other packages would choose to leave the error handling to python, which means there would be a standard KeyError message only.

And we can remove the SSH tunnel message if you find it distracting (make it a DEBUG level).

It simply doesn't make sense to have these type of error and confuse the users. I strongly suggest to have clear error messages, catch those errors early and exit without the python errors.
Error messages should be helpful. And here we don't really have an exception, it is a file not found, we can handle it and exit.

On the issue of the 'data' directory: I don't want to start a discussion about directory names, but I think it would have been clearer if it wouldn't be called data but maybe configuration?

Seeing the full output of this application, I get now the following error at the end.
Is the connection the DB lost when running an application which takes several minutes?
This is running from home on my laptop.
INFO::ray_tracing(l227)::simulate::Simulating RayTracing for offAxis=0.0, mirror=170
INFO::simtel_runner(l173)::run::Running (1x) with command:/workdir/sim_telarray/sim_telarray/bin/sim_telarray -c /workdir/external/output/simtools-output/derive_mirror_rnda/model/CTA-North-MST-FlashCam-D-prod4_derive_mirror_rnda.cfg -I../cfg/CTA -C IMAGING_LIST=/workdir/external/output/simtools-output/derive_mirror_rnda/ray-tracing/photons-North-MST-FlashCam-D-d0.0-za20.0-off0.000_mirror170_derive_mirror_rnda.lis -C stars=/workdir/external/output/simtools-output/derive_mirror_rnda/ray-tracing/stars-North-MST-FlashCam-D-d0.0-za20.0-off0.000_mirror170_derive_mirror_rnda.lis -C altitude=2150.0 -C telescope_theta=20.0 -C star_photons=10000 -C telescope_phi=0 -C camera_transmission=1.0 -C nightsky_background=all:0. -C trigger_current_limit=1e10 -C telescope_random_angle=0 -C telescope_random_error=0 -C convergent_depth=0 -C maximum_telescopes=1 -C show=all -C camera_filter=none -C focus_offset=all:0. -C camera_config_file=single_pixel_camera.dat -C camera_pixels=1 -C trigger_pixels=1 -C camera_body_diameter=0 -C mirror_list=/workdir/external/output/simtools-output/derive_mirror_rnda/model/CTA-single-mirror-list-North-MST-FlashCam-D-prod4-mirror170_derive_mirror_rnda.dat -C focal_length=3214.0 -C dish_shape_length=1607.0 -C mirror_focal_length=1607.0 -C parabolic_dish=0 -C mirror_align_random_distance=0. -C mirror_align_random_vertical=0.,28.,0.,0. /workdir/sim_telarray/run9991.corsika.gz 2>&1 > /workdir/external/output/simtools-output/derive_mirror_rnda/ray-tracing/log-North-MST-FlashCam-D-d0.0-za20.0-off0.000_mirror170_derive_mirror_rnda.log 2>&1
INFO::ray_tracing(l227)::simulate::Simulating RayTracing for offAxis=0.0, mirror=171
client_loop: send disconnect: Broken pipe
Traceback (most recent call last):
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1514, in _retryable_read
    server = self._select_server(
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1346, in _select_server
    server = topology.select_server(server_selector)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/topology.py", line 244, in select_server
    return random.choice(self.select_servers(selector,
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/topology.py", line 202, in select_servers
    server_descriptions = self._select_servers_loop(
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/topology.py", line 218, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: localhost:27018: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 6200d849ebe36888bd560be0, topology_type: Single, servers: [<ServerDescription ('localhost', 27018) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27018: [Errno 111] Connection refused')>]>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 311, in <module>
    meanD80, sigD80 = run(newRnda)
  File "gammasim-tools/applications/derive_mirror_rnda.py", line 241, in run
    ray.simulate(test=False, force=True)  # force has to be True, always
  File "/workdir/external/gammasim-tools/simtools/ray_tracing.py", line 232, in simulate
    simtel = SimtelRunnerRayTracing(
  File "/workdir/external/gammasim-tools/simtools/simtel/simtel_runner_ray_tracing.py", line 119, in __init__
    self._loadRequiredFiles()
  File "/workdir/external/gammasim-tools/simtools/simtel/simtel_runner_ray_tracing.py", line 154, in _loadRequiredFiles
    "# configFile = {}\n".format(self.telescopeModel.getConfigFile())
  File "/workdir/external/gammasim-tools/simtools/model/telescope_model.py", line 525, in getConfigFile
    self.exportConfigFile()
  File "/workdir/external/gammasim-tools/simtools/model/telescope_model.py", line 502, in exportConfigFile
    self.exportModelFiles()
  File "/workdir/external/gammasim-tools/simtools/model/telescope_model.py", line 496, in exportModelFiles
    db.exportModelFiles(parsFromDB, self._configFileDirectory)
  File "/workdir/external/gammasim-tools/simtools/db_handler.py", line 265, in exportModelFiles
    self._writeFileFromMongoToDisk(
  File "/workdir/external/gammasim-tools/simtools/db_handler.py", line 685, in _writeFileFromMongoToDisk
    fsOutput.download_to_stream_by_name(file.filename, outputFile)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/gridfs/__init__.py", line 910, in download_to_stream_by_name
    for chunk in gout:
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/gridfs/grid_file.py", line 802, in next
    chunk = self.__chunk_iter.next()
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/gridfs/grid_file.py", line 755, in next
    chunk = self._next_with_retry()
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/gridfs/grid_file.py", line 747, in _next_with_retry
    return self._cursor.next()
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/cursor.py", line 1238, in next
    if len(self.__data) or self._refresh():
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh
    self.__send_message(q)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/cursor.py", line 1044, in __send_message
    response = client._run_operation(
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1424, in _run_operation
    return self._retryable_read(
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1531, in _retryable_read
    raise last_error
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1525, in _retryable_read
    return func(session, server, sock_info, secondary_ok)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1420, in _cmd
    return server.run_operation(
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/server.py", line 114, in run_operation
    reply = sock_info.receive_message(request_id)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/pool.py", line 753, in receive_message
    self._raise_connection_failure(error)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/pool.py", line 751, in receive_message
    return receive_message(self, request_id, self.max_message_size)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/network.py", line 215, in receive_message
    data = _receive_data_on_socket(sock_info, length - 16, deadline)
  File "/conda/envs/gammasim-tools-dev/lib/python3.8/site-packages/pymongo/network.py", line 293, in _receive_data_on_socket
    raise AutoReconnect("connection closed")
pymongo.errors.AutoReconnect: connection closed
INFO::db_handler(l187)::_closeSSHTunnel::Closing SSH tunnel(s)

It is working normally with me, after I fixed a bug.

Let me check this on the WGS. At home on my laptop, it shows this error after a while. Main reason is probably some time out of the ssh connection.

If that's the case, I will have to modify the tunnel or DB connection parameters so that it doesn't happen. Let me know and I will try to recreate then.

The error message is
pymongo.errors.AutoReconnect: connection closed

Looking around, it seems worth trying to keep the connection alive, see e.g. here

It could also be an issue of trying to open a second connection when running the second mirror (instead of using the original connection). I will try to reproduce and debug.

Note that on your laptop, the kerberos ticketing is probably working (it stopped working probably after an update on mine, and I didn't have time to fix it). Maybe this is the difference - I would suggest not to dig further.
Anything improving efficiency is good in my opinion.

Actually no, the token did not work. I think that the update to Monterey modified the way OSX saves the krb token. Spent 15 minutes trying to fix it and gave up. So our setup was equivalent.
However, I can imagine that keeping a tunnel open for 2 hours could cause an issue. We can both try to extend the timeout and to make the code more efficient. The former requires help from the DB manager. The latter I will try to figure out in the next few days (unless @RaulRPrado has a reason why it cannot be done).

Couldn't recreate, neither from the WGS nor from my laptop using the usual dev container. Running on the WGS took about 10 minutes, running on the laptop took significantly longer (1.5-2 hours). In neither case did I get this issue.
However, following your error message, I do see some strange behaviour in the code. If I understand correctly (@RaulRPrado can correct me if I am wrong), every time we run the ray tracing on one mirror, we read the model from the DB and export a config file. Not sure why we do that for every mirror. I assume the entire config file does not change but instead we change just one/two parameters in it. Why don't we export the file once and then, if necessary, edit the exported one prior to each run? Alternatively, we not export the model we already read once and have in memory (i.e., dict)?
I am pretty sure that reading from the DB for every run makes the execution much slower, especially when running from home. Should we open an issue to improve this behaviour?

That should be easy to fix. I will work on it.

I will create another issue for that, so you can close this one.

OK, I am closing this. The underlying issue of prolonging the timeout of the DB might still be there, but if we fix this issue we might never encounter it again and maybe increasing the timeout unnecessarily isn't a good idea.


ci(perf): Disable unused .NET native for uwp

Improves build speed by removing UWP .NET native build

@mergifyio backport release/stable/3.1 legacy/3x


Needextract does not work with images

Describe the bug
We want to add images to the content of needs and reference these needs with needextract. Unfortunately, the images are not correctly parsed when extracting the need.
I assume it is related to https://github.com/useblocks/sphinx-needs/pull/794. However, the PR only solves the problem if images are located in source/_images. Where the images will also be located in build.
To Reproduce
Steps to reproduce the behavior:

Create projectA:mkdir projectA
cd projectA
sphinx-quickstart

I choose to separate source and build folder during the build, i.e., Separate source and build directories (y/n) [n]: y
Configure projectA by adding the following lines to projectA/source/conf.py:extensions = ["sphinx_needs"]

Add an image to the source folder, i.e.. projectA/source/image.png.
Create a requirement by adding the following lines in projectA/source/index.rst:.. need:: NeedA
   :id: NEEDA

Link the image with needextract:.. needextract:: NEEDA

Expected behavior
The requirement NEEDA appears twice and contains the used image twice.
Actual behaviour
For the first need, displays the image, the second need displays the path in source/index.rst

Before using the most recent sphinx-needs release, the path was _images/image.png, i.e., the image copy in build. This broken as soon as the rst file was not on the source level but in a subfolder.
Setup
We are using python Python 3.8.13. And the following version of sphinx/sphinx-needs:
sphinx-needs                  1.1.0
Sphinx                        5.3.0

I hope the description helps to reproduce the issue :)
The issue template was copied from labs-zola.

Hi @lehaas,
First of all, I would like to thank you for the issue report.
I read through the description you gave on how to reproduce the issue and would like to clarify some things.

We want to add images to the content of needs and reference these needs with needextract. Unfortunately, the images are not correctly parsed when extracting the need.
I assume it is related to https://github.com/useblocks/sphinx-needs/pull/794. However, the PR only solves the problem if images are located in source/_images. Where the images will also be located in build.

The statement above is not entirely accurate because if you have an image directive (or an image) added to the content of a need directive, the images are copied and stored in the _images folder inside your build or _build directory.
Example
Setup
We are using python Python3.9.13. And the following version of sphinx/sphinx-needs:
sphinx-needs                  1.1.0
Sphinx                        5.3.0

Project File Structure

Create projectA:
mkdir projectA
cd projectA
sphinx-quickstart

I choose not to separate source and build folder during the build, i.e., Separate source and build directories (y/n) [n]: n but if you choose yes it should work.

Configure projectA by adding the following lines to projectA/conf.py:
extensions = ["sphinx_needs"]

Add an image to the project folder, i.e. projectA/emoji.png.

Create a requirement by adding the following lines in projectA/index.rst:
.. need:: NeedA
   :id: NEEDA

   .. image:: emoji.png

Reference the need above with needextract:
.. needextract:: NEEDA

Expected behaviour
The need NEEDA appears twice and contains the used image twice.
Actual behaviour
The need NEEDA appears twice and contains the used image twice.

@iSOLveIT okay, maybe I broke down the example to far and it works now.. I was quite sure it did not work on my end. But you are right. That was my fault.
Let me give it another try, this time defining and linking the need in different files, as I actually want to use it and there it breaks on my end:
To Reproduce
Steps to reproduce the behavior:

Create projectA:mkdir projectA
cd projectA
sphinx-quickstart

I choose to separate source and build folder during the build, i.e., Separate source and build directories (y/n) [n]: y
Configure projectA by adding the following lines to projectA/source/conf.py:extensions = ["sphinx_needs"]

Move to the projectA/source.
Add an image to the source folder, let us call it image.png.
Create a subfolder of source contain a file for the need definition subfolder/image_need.rst and add the following content :

Page
====

.. need:: NeedA
   :id: NEEDA

   .. image:: /image.png

Link the image with needextract in `index.rst:

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   subfolder/image_need.rst

.. needextract:: NEEDA

Expected behavior
The requirement NEEDA appears in both files and each contains the used image.
Actual behaviour
For the first need, displays the image, the second need displays the path in subfolder/image_need.html

In index.html the image link is broken:

Setup
We are using python Python 3.8.13 or 3.9.13. And the following version of sphinx/sphinx-needs:
sphinx-needs                  1.1.0
Sphinx                        5.3.0

Hi @iSOLveIT ,
thanks for the answer. I am sad to hear that absolute paths don't work yet. Please let me know as soon as it is implemented.
On the relative paths: I don't see that relative paths solve the issue. In your code example, US_002 was never used in a need_extract. If I add it to index.rst, again the image is not visualized correctly:

Here is the zipped version:
issue-818.zip
Btw, I am using make html to build the project.

Hi all,
thanks for this detailed discussion. It helped me a lot to div deeper into the root cause of the problem.
First of all, the current implementation of needextract and images is nearly totally broken.
I have extended our test case for different scenarios, how images are referenced.
You can find it here: https://github.com/useblocks/sphinx-needs/tree/needextract_image/tests/doc_test/doc_needextract
The test-doc is creating needs incl. images with absolute and relative paths.
Images are stored on root or in a subfolder. And the need is referenced by needextract in the same file or in a file outside need-folder.
Here are the results of what is working:

As you see, only images stored inside /_images and referenced by a relative path are supported. Everything else does not work.
I see 2 possible error sources:

Sphinx-Needs is handling the image-node creation wrong. But as far as I know, we currently do nothing with paths and co. of referenced needs.
2 The Sphinx image handling is not triggered for the created image-nodes after needextract has copied the needs.

I'm afraid it is point 2, as Sphinx is doing the image-handling far before Sphinx-Needs is executing the needextract code.
Sphinx image handling is directly executed when an image directive is detected. So in the run() function of the image directive.
Not sure, how we can reinvoke this behavior after needextract got executed.

Mhh the content get copied inside needextract via the create_need() function in layout.py.
https://github.com/useblocks/sphinx-needs/blob/56b4835ef6b3044e9bbde59681998bfa247fb549/sphinx_needs/layout.py#L47
So this line is reusing the complete content node-tree, which contains everything like text, table and maybe images-nodes.
It's using completely the same nodes as the original need, so I'm very confused why a reused image-node does not behave the same way.
Especially, why are relative paths working, but no absolute paths?
I also tried to create a copy via  node_inner = needs[need_id]["content_node"].deepcopy() instead of using the same referenced nodes.
But no change in the behavior.

@danwos thanks a lot for the informations. We discovered one interesting point: Using sphinx-needs==1.0.3, images work just fine in need_extract. Not sure if there were any huge changes on your end that might have changed it. Maybe that can help you to ping down the error.

Especially, why are relative paths working, but no absolute paths?
I don't agree with you that relative paths are working in sphinx-needs==1.1.0. On my end the table should as follows:

Using relative paths, subfolder structure and storing the image in _images does not work on my end. See the following code:
issue-818.zip

I make a quick check, if 1.0.3 behaves different as 1.1.0
And this is the result of index.html  under tests/doc_test/doc_needextract:

Now no needextract from a need inside subfolder/file_1.rst works. So my table and also the one from @lehaas do  not cover the new result. Can't say why.
These are the changes between 1.0.3 and 1.1.0:
https://github.com/useblocks/sphinx-needs/compare/1.0.3...1.1.0
One of the major changes was, that we do not save any docutils nodes from the doctree anymore, as this would keep the complete doctree in memory.
Instead, we reference it by ID or create a copy, which is not part of the doctree anymore.
The last one happens also with the need-content node.
See this line
Also the event-handling was reimplement a little different, but the order should be the same.
See this line
I will revert some changes from 1.1.0 back to 1.0.3 to detect the source of the problem.

@danwos: I noticed another problem with needextract in sphinx-needs 1.1.0.
It causes a sphinx error in our documentation project. I'll open another issue for it.

@danwos I found another bug with images in v1.0.3 and v1.1.0. If needs are referenced with needextract in multiple locations, the image link only works in the first reference in v1.0.3. The issue is further described in the PR. See https://github.com/useblocks/sphinx-needs/pull/837.
Feel free to add it to your PR if you don't want to add a new folder.

Thanks all for testing and the detailed error reports.
I found a fix and integrated it already into PR #825.
Most important part is this one, which recalls Sphinx internal Transformators for need-content copied by needextract.
https://github.com/useblocks/sphinx-needs/blob/869e6c2330a5482e0291a4ca57fdfe64dcfa3bb3/sphinx_needs/layout.py#L68-L71
This fixes also the multiple reference problem reported shortly by @lehaas.
Haven't found the time to clean the PR and finalize the tests, but will do it soon.
Any tests with your projects are welcome ;)


Task @InputFile of type Provider<File> should behave like a normal File type

Provider<File>, most often RegularFileVar, can be used as file inputs and should behave like a normal File type.
Expected Behavior
Given the following build script:
class CustomTask extends DefaultTask {
    @InputFile final RegularFileVar input

    CustomTask() {
        input = newInputFile()
    }

    @TaskAction void doSomething() {
        throw new GradleException("This task shouldn't execute")
    }
}

task customTask(type: CustomTask)

Executing customTask throw InvalidUserDataException with the message No value has been specified for property 'input'.
Current Behavior
Currently, it throws IllegalStateException with the message No value has been specified for this provider. This message is unintuitive and doesn't provide enough information on the error.
Context
See this slack conversation.

@lacasseio do you still plan to have this in 4.3?

@lacasseio this is still marked 4.3, but I see no activity on the PR. Can you please update the milestone or merge today if it's good to go?

My bad, this work was differed to later, probably 2 weeks from now.

This is fixed now, but in a slightly different and simpler way than in the PR.


Make script not check for installation of requirements each run

Maybe make the user install the requirements.txt themselves... unless there are any other ideas?

In the fix for issue #11 I partly address this; I will be implementing a change where the script, on first run, will write the user's username to the script.config file, so that way all they have to type in is either their password or PIN. When I get that PR in, we can change the script to check if that variable is already set; if it is, then we won't check requirements. If not, then we can check requirements

@MisterWhopper Good to know are you almost ready to send in that PR?


Activity Feed Auto-Scroll Occasionally Not Working

Plugin version: 6.0.1 Stable
Chrome/Opera/Chromium version: Chrome 64.0.3282.186 (also happens on another computer running Chrome 63.0.3239.132)
OS version: Windows 10

Steps to reproduce:

Scroll down activity feed (https://www.strava.com/dashboard/)
Occasionally auto-scroll fails and the "More" button appears. This appears to be random, but is often on the first page load (https://www.strava.com/dashboard/following/) and when the url is (https://www.strava.com/dashboard/following/90)
Note: Have tried in incognito mode (with the plugin enabled), disabling all other plugins, and using another computer. Auto-Scrolling works correctly when disabling the plugin.

Actual Behavior:

What happened?
Do you have javascript errors logs in browser console?

Press F12 to see developer console and javascript logs

No install or update detected
localStorageMustBeCleared is now true
A previous sync exists on Fri Mar 02 2018 01:43:36 GMT-0600 (Central Standard Time)
Do not auto-sync. Last sync done under than 60 minute(s) ago

Expected Behavior:

What did you expect to happen?
The feed should auto-scroll until reaching the end without having to click the "more" button.

Media attachement:

You can upload/attach any images to this ticket.

2018-03-02 01_50_58-dashboard _ strava
Providing your stravistix synced athlete history can be usefull to solve the problem

To export you history click on "sync button" (top right and corner), then "Backup history". You can zip the file and upload it here.

2018.3.2-1.41_v6.0.1.history.zip

I were not able to reproduce this. Still experiencing it? I let you close the ticket or not.
Ty


Use a real clock for the reporting plugin instead of the stub vers…

…ion of halCommonGetInt32uMillisecondTick
Problem
The reporting plugin use a stub version of halCommonGetInt32uMillisecondTick which always return 0.
I did not fix halCommonGetInt32uMillisecondTick because I believe the goal is to remove this method at the end since the platform layer already act as the HAL layer.
Summary of Changes

Use a platform specific clock to get the elapsed time in MS

I just pushed an update to the PR that basically does the same thing for the IAS Zone server cluster.


Refactor sender middleware

This PR fixes some issues in the outbound middleware design.
Now it is possible for sender middleware to read current message, current message headers and modify outbound message headers.
This is one step closer to full tracing support.
This is related to #127

Seems awkward.  So this puts the burden on each caller of the actor to setup the sender middleware instead of having it part of the props on the actor?

No, the actor can also have it, but that is for messages going out of the actor.
If you have tracing, the caller owns the context.
Eg your rest endpoint need to extract tracing info and pass forward

I guess, thinking about it more, all you need to do is have an extension method that creates the root context and overload Tell to take the sender middleware 👍

"extension method" on what?
We are going from the outside world into the actor world, and we want to propagate contextual information, e.g. tracing values.
[Console app]  --- tell --> [Actor]
[Root Context] -----------> [Actor Context]
[Sender Middleware] ------> [Receive Middleware]
[Message Headers Out] ----> [Message Header In]

I've applied some refactoring on the API level.
RootContext -> ActorClient
Context.MessageHeader -> Context.Headers
RootContext felt a bit confusion and didn't quite communicate what it was.
ActorClient is what it sounds like, it is a client to interact with Actors.
The reason those exists, is if you want to apply middleware and headers between your non-actor code into the world of actors.

I mean, couldn't you have an extension method off of PID that looked something like: public static void Tell(this PID pid, object message, Func<Sender, Sender> middleware) and inside that method create the ActorClient and send?
i guess you'd usually want to reuse Actor clients.  Can I abuse message headers to have an expected reply message type? :)

Can I abuse message headers to have an expected reply message type?

Headers are much like HTTP headers, just KVP of strings. I guess you could pass the expected response type name that way.
But then again, that would probably just be easier to just do that in the message itself.

btw. is there anything that needs to be done here before merging?


Add COG overview information to info?

Add it here? https://github.com/cogeotiff/rio-tiler/blob/702b7c973b533c040f00e8eb73baceda5fcf292a/rio_tiler/io/cogeo.py#L178-L220
Would allow an application to deduce appropriate zoom levels from titiler's /cog/info response.

Would allow an application to deduce appropriate zoom levels from titiler's /cog/info response.

@kylebarron I'm not sure to get this. the minzoom/maxzoom is already in the info() response but the TiTiler /info endpoint ignores it https://github.com/developmentseed/titiler/blob/master/titiler/endpoints/factory.py#L170 because the zooms are linked to a TMS grid which is not an input of the /info endpoint
That's said, I'm ok adding more low level dataset info like overview level

I added the overview decimation levels in https://github.com/cogeotiff/rio-tiler/pull/387, I think this is enough for now!


Update JH README page

Update the link of the GitHub Actions build status badge, acknowledge the original source of the code, update all contents to match our own project.

Codecov Report

Merging #24 (5742b3d) into master (d1a0917) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master      #24   +/-   ##
=========================================
  Coverage     72.15%   72.15%
  Complexity      399      399
=========================================
  Files            70       70
  Lines          1232     1232
  Branches        125      125
=========================================
  Hits            889      889
  Misses          311      311
  Partials         32       32

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Ternary Search Added & Codeforces api also added

Description

Please provide a summary of your project or the changes related to the issue. Include relevant motivation and context to help us understand the purpose and impact of your work.
List your tech stack and any dependencies that are required for this change. This information will help us review and integrate your contributions effectively.

Type of change

[x] New Project
[x] New Feature
[x] Bug fix

Issue number:

Checklist:

[X] I have included a README.md file for my project clearly explaining its purpose, functionality, and usage instructions (applicable for new projects, not bug fixes).
[X] My code adheres to the style guidelines outlined by this project.
[X] I have thoroughly reviewed my code to identify and address any potential errors or inconsistencies.
[X] I have added clear and concise comments to my code, especially in sections that might be challenging to understand at first glance.
[X] I have reviewed and followed the project's Contributing Guidelines to ensure my contribution aligns with the project's requirements.

Your PR conflict our guidelines but I am accepting that because it is the last moments of Hacktoberfest


Ellipsis constant is defined as a function?

Unsure if this is a valid report yet, but PHP 7.2 (current master branch of PHP) reports an error in my builds due to this line, copied for reference:
/** Define unicode ellipsis constant */
define('Functional\\…', …());

Apparently, that defines a constant that is actually a function, and that raises a WARNING in PHP 7.2.

Hmmm, it looks like I can no longer register a resource as a constant 👎
That was such a cute trick to mark a parameter as partial but it looks like I have to find something better

Attempt to fix this issue: 7633c9f235a8946bb9e36f994010e018806f8ce8

R U TROLEING THE ENGINE?
Marco Pivetta
http://twitter.com/Ocramius
http://ocramius.github.com/
On Mon, May 8, 2017 at 11:12 AM, Lars Strojny<EMAIL_ADDRESS>wrote:

Attempt to fix this issue: 7633c9f
https://github.com/lstrojny/functional-php/commit/7633c9f235a8946bb9e36f994010e018806f8ce8
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/lstrojny/functional-php/issues/138#issuecomment-299814756,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJakOzABm6JTjh3WyGhAAolTJTnqxWnks5r3txpgaJpZM4NQgv_
.

@Ocramius I try but without any luck. So random_bytes() it is.
See 5ed5ee0

Fixed in latest master. Thanks @Ocramius !

@lstrojny thanks!


feat: Squadcast

Summary
Basic overview of squadcast integration. New Relic's alerting webhooks triggering squadcast workflow incidents.

Pre merge checklist

[x] Did you include a Data source and Documentation reference?
[x] Are all documentation links publicly accessible?
[x] Did you check your descriptive content for voice and tone?
[x] Did you check your descriptive content for spelling and grammar errors?
[x] Did you review your content with a subject matter expert? (e.g. a Browser agent quickstart is reviewed with a member of the Browser Agent team)

Hi @RamanaReddy8801 , There is no datasource for this integration because New Relic is the datasource. The documentation shows how to create a NR webhook to send data to squadcast.


Add type for match doc

{ thanks: true }

😂👍🏼
On Thu, 23 Mar 2017 at 18:33, Tim Dorr<EMAIL_ADDRESS>wrote:

Merged #4822 https://github.com/ReactTraining/react-router/pull/4822.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ReactTraining/react-router/pull/4822#event-1012890537,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAyEp34heu_0EjG35JdNcQRUcj1YNouEks5roqzJgaJpZM4Mm-K_
.


fix: Gallery page

Related Issue
Closes: #533

Description

Removed extra code
Removed breadcrumps~fixed head section of gallery page
Fixed footer and aligned it similar to overall design of the page

Screenshots
https://github.com/Eduhub-Community/Eduhub-Community.github.io/assets/100852245/19446a8b-bce2-4c12-aae5-72880ece44d8

Checklist

[X] My code adheres to the established style guidelines of the project.
[X] I have included comments in areas that may be difficult to understand.
[X] My changes have not introduced any new warnings.
[X] I have conducted a self-review of my code.

@AshAman999 Please review and merge this pr.
I hope It will be level 2 as It multiple files had to be fixed to solve the issue.

@AshAman999 ??

@AshAman999 Please review and merge this pr. I hope It will be level 2 as multiple files had to be fixed to solve the issue.

Can you please merge it as well?


Support gpt-3.5-turbo-0125, gpt-3.5-turbo-instruct, and gpt-4-turbo-0125-preview

This PR kicks off support for the gpt-3.5-turbo-0125, gpt-3.5-turbo-instruct, and gpt-4-0125-preview models.
It also sets the default gpt-3.5-turbo model to gpt-3.5-turbo-0125.

Thank you very much for your contribution, but I have optimized the mapping between models and prices in v1.3.2 (After that just maintain the file src/pricing.ts)
So these submissions cannot be merged.


Query param route predicate - extension of QueryRoutePredicateFactory

I tried to generalize the QueryRoutePredicateFactory so that it could use a bean of type Predicate instead of being limited to a regexp. Test classes are also implemented.

I'd prefer this in one class as having multiple query predicates is confusing.

@spencergibb you are suggesting to merge the two classes into one single predicate factory, that can manage both cases, right? I can do it, paying attention not to introduce regressions on the QueryRoutePredicateFactory.

@polifr yes, since this will only be available thru the java dsl, it should be ok. the regex can be retrofit to a Predicate.

@spencergibb I modified the QueryRoutePredicateFactory class to unify the regexp and predicate approach. Let me know if this is what you meant, so that I can proceed with adjustments on junit tests and cleaning of the previous proposal.

Yes, you can now remove QueryParamRoutePredicateFactory

Ok, I  removed the QueryParamRoutePredicateFactory and QueryParamRoutePredicateFactoryTests and extend the current QueryRoutePredicateFactoryTests to cover the new options. I also fixed the checkstyle errors that made the build fail.

Hi @spencergibb , is there something that stills blocks this pull request? I can't see any change request that has to be done.

There was a format error in the header of the test class, I fixed it.


Perform object storage test at startup

Test object storage at startup to fail fast instead of throwing EIO in the middle of user's workload.

It's slow to test, which will slow down the mount.
The AK/SK is verified in format/config, which should be enough.

format only verifies the AK/SK but cannot check user's environment. A typical scenario is when a node cannot connect to OSS for various reasons, such as DNS issues or VPC network isolation problems. In such cases, users may encounter EIO errors in the middle of his task.
If we can perform some cheap self-checks at startup to expose errors early, we can avoid expensive user-side errors.

It could be useful in some cases, should be enabled by a flag.

Do you have a suggestion for this flag's name?

--check-storage?

Updated. Please review, thanks!


Update Go deps

This PR pulls in changes from https://github.com/libsql/sqlite-antlr4-parser/pull/4, which fixes the import.
In another project I work on, pulling github.com/tursodatabase/libsql-client-go as a dependency causes Go to get tripped up (the underlying issue was fixed in https://github.com/libsql/sqlite-antlr4-parser/pull/4):
go<EMAIL_ADDRESS>parsing go.mod:
	module declares its path as: github.com/antlr4-go/antlr/v4
	        but was required as: github.com/antlr/antlr4/runtime/Go/antlr/v4
	restoring<EMAIL_ADDRESS>
Ran:
go get -u ./... && go mod tidy

I think some dependency got messed up? It is breaking go vet in the CI
Error: ../../../go/pkg/mod/github.com/libsql/sqlite-antlr4-parser@v0.0.0-20240323153644-5c63f680a14e/sqliteparserutils/utils.go:57:42: cannot use antlr.NewInterval(currentIntervalStart, previousToken.GetTokenIndex()) (value of type antlr.Interval) as *antlr.Interval value in argument to append
Error: ../../../go/pkg/mod/github.com/libsql/sqlite-antlr4-parser@v0.0.0-20240323153644-5c63f680a14e/sqliteparserutils/utils.go:66:41: cannot use antlr.NewInterval(currentIntervalStart, previousToken.GetTokenIndex()) (value of type antlr.Interval) as *antlr.Interval value in argument to append
Error: ../../../go/pkg/mod/github.com/libsql/sqlite-antlr4-parser@v0.0.0-20240323153644-5c63f680a14e/sqliteparserutils/utils.go:71:57: cannot use stmtInterval (variable of type *antlr.Interval) as antlr.Interval value in argument to tokenStream.GetTextFromInterval

I think this is coming from github.com/libsql/sqlite-antlr4-parser directly,
# github.com/libsql/sqlite-antlr4-parser/sqliteparserutils
sqliteparserutils/utils.go:57:42: cannot use antlr.NewInterval(currentIntervalStart, previousToken.GetTokenIndex()) (value of type antlr.Interval) as *antlr.Interval value in argument to append
sqliteparserutils/utils.go:66:41: cannot use antlr.NewInterval(currentIntervalStart, previousToken.GetTokenIndex()) (value of type antlr.Interval) as *antlr.Interval value in argument to append
sqliteparserutils/utils.go:71:57: cannot use stmtInterval (variable of type *antlr.Interval) as antlr.Interval value in argument to tokenStream.GetTextFromInterval

Ah, I see the problem I think this should not be a pointer
https://github.com/libsql/sqlite-antlr4-parser/blob/5c63f680a14ef6d0de8e58ffba89b17f4e2b2dc7/sqliteparserutils/utils.go#L20

@avinassh This should be ready to trigger the CI workflow.


Bugfix: include is_vip and profile in get_detailed_player_info

Hello,
get_detailed_player_info did not override the default false for is_vip and null for profile with the correct values.
This PR fixes that and removes the workaround in get_team_view to not fetch the profiles twice. This also fixes the wrong value is_vip in get_detailed_players.
Thank you!

The last commit fixes the not showing time in the Team View Frontend Site.


[Question]: Invoke a codeunit for loading additional data / manipulating the container

Question
In our current Azure DevOps Pipeline-setup we use a custom codeunit to manipulate the data of a container / load new data in order to run our tests. The data is stored in our repo and depends on the currently version of the app under test.
We did not find an action which allows us, to invoke a codeunit inside a container in AL-Go. Is this possible?

Our recommendation is to use configPackages.
If you cannot use that, our recommendation is to have an app, which populates demo data on install and then have a dependency on this app from the other test apps (ensuring that it is installed first).
If you cannot use that, you can place a file called ImportTestDataInBcContainer.ps1 in your .AL-Go folder in the project, like this:
Param([Hashtable]$parameters)

$country = Get-BcContainerCountry -containerOrImageName $parameters.containerName
$prop = "configPackages.$country"
if ($settings.Keys -notcontains $prop) {
    $prop = "configPackages"
}
if ($settings."$prop") {
    Write-Host "Importing config packages from $prop"
    $settings."$prop" | ForEach-Object {
        $configPackage = $_.Split(',')[0].Replace('{COUNTRY}',$country)
        $packageId = $_.Split(',')[1]
        UploadImportAndApply-ConfigPackageInBcContainer `
            -containerName $parameters.containerName `
            -companyName $settings.companyName `
            -Credential $parameters.credential `
            -Tenant $parameters.tenant `
            -ConfigPackage $configPackage `
            -PackageId $packageId
    }
}

This is the override provided by AL-Go for GitHub when using configPackages - but you could really install an app and run a codeunit in this script as well.

Thank you!


Import fails to find pacakge that exists

Really loving gore so far, but found an odd bug.
 Is this a bug in gopter? I got the package name from the Invalidconcat example here: https://godoc.org/github.com/leanovate/gopter/prop#example-package--Invalidconcat

You can avoid error by installing github.com/leanovate/gopter/gen.
% go get github.com/leanovate/gopter/gen

I suppose this caused bygo/importer. importer.Default().Import(path) which is used in gore for importing package finds only under $GOPATH/pkg. go get github.com/leanovate/gopter builds only $GOPATH/pkg/your_architecture/github.com/leanovate/gopter.a and does not build $GOPATH/pkg/your_architecture/github.com/leanovate/gopter/gen.a. So you got such error.

I'm confused though - I can take the example gopter code and go run it fine. So what is run/build doing that gore is not?

go run compiles source code and run it. While :import of gore loads package object file(*.a file on Linux system), it does not compile source code and not generate object file. :import commands raises error if there is no object file. So you need to generate object file before using :import command.

Ahh right, that makes sense, thanks! I'm still kind of new to go. I managed to fix it via go install github.com/leanovate/gopter/gen
Is it possible that gore could notice that case and either run the install for me, or print a more understandable error message? I guess anything is possible, is it practical?

Yes it is possible. Code is here.
https://github.com/motemen/gore/blob/603106957dbb1f054a60be39c4bc0c84cc060e44/commands.go#L84-L88

Thanks both, yes it is the current behavior of gore to  require :imported packages to be intalled, which causes such confusion. I think this could be fixed.

I was wrestling with this for a while and could only get it to work by go getting the package I needed and then running gore from my home directory:
zac@lenovo ~ % go get github.com/aws/aws-sdk-go/aws/session
zac@lenovo ~ % gore
gore version 0.3.0  :help for help
gore> :import github.com/aws/aws-sdk-go/aws/session
gore> session.NewSession
(func(...*aws.Config) (*session.Session, error))(0x6efdb0)
gore>

@motemen I'd be happy to add this the README if you could provide a bit of context as to why this is the case?


Make Space Data Explorer

For those moments when Earth just isn’t exciting enough.
Uses NASA’s API to display daily space pictures, Mars Rover images, or astronomy facts. It could even display real-time data on asteroids near Earth.

Hey sumanth, can u assign this to me ?


Use scout multiplier from user.cfg instead of hard coded

There are a couple of hard coded things in the BTB manager which can also be read from the user.cfg, i.e. the scout_multiplier.
Closes #88

@all-contributors please add @sydekumf  for code


Delete downloaded files from disk

Hello!
It would be nice if podboat could remove the files on disk its list entries refer to, instead of just the list entries themselves. At the moment, to keep the downloads folder from ballooning, the user has to regularly go into the folder and delete old podcasts, which can become a chore when there's a huge number of them both already downloaded but not yet listened to and also already listened to and removed from podboat's list.

Hi! Thank you for the suggestions. This makes perfect sense; I'll keep it. However, Podboat doesn't get any attention nowadays, so I'm not sure when this would be implemented.

Are there any plans as to when the next release incorporating this change will be made?

We release every three months, so around 20th of March, @hfel.


HLint doesn't understand that identifiers with the same name can have different types and values, which can introduce subtle runtime bugs

Consider this program:
import Data.Maybe (isJust)

threeTenths :: RealFrac a => Maybe a
threeTenths = if 1/10 + 1/5 == correctAnswer then Just correctAnswer else Nothing
  where correctAnswer = 3/10

main :: IO ()
main = print $ isJust (fmap fromRational threeTenths)

It prints True. Running hlint on it will suggest using isJust threeTenths instead of isJust (fmap fromRational threeTenths). However, implementing that suggestion will cause it to instead print False.

Hmm. It's not good that this program gives different results after the refactor. But I'd also argue that this program is pretty dodgy in the first place - it's only wrong because of equality on floating point values (which is dubious in the first place) and non-conforming instances (e.g. Double). Therefore, I'm inclined to document it in the README as a curiosity, rather than change HLint to deal with it.

Don't close this just yet — I have a feeling I can get it to break without the abomination that is Double.

Sure, also open to other viewpoints on this issue.

Okay, got one. This one will switch from False to True, without using equality on floating point values, and I think without exploiting non-conforming instances:
import Data.Maybe (isJust)
import Data.Word (byteSwap16)

threeHundred :: Integral a => Maybe a
threeHundred = if (300 * 400) `div` 400 == correctAnswer then Just correctAnswer else Nothing
  where correctAnswer = 300 * (400 `div` 400)

main :: IO ()
main = print $ isJust (fmap byteSwap16 threeHundred)

Is this exploiting integer wrap around? That's also undefined in Haskell, I believe.

The only data type that's "overflowing" is a Word16, and from Data.Word:

All arithmetic is performed modulo 2^n, where n is the number of bits in the type.

So I think this is completely kosher.

Hmm, that's annoying. The code still seems a little funky/contrived. Is the isJust/fmap rule the only one that triggers this issue? If so, I'm inclined to kill it to be on the safe side (it's not a big deal). Or does it hit a lot of other rules? I guess only rules where we manage to "delete" a whole piece of work, which is pretty rare?

I think it does hit a lot of other rules.

Here's an example that has nothing to do with isJust or fmap, or even Maybe or any Functor, and affects a rule that doesn't "delete" a whole piece of work:
import Data.Maybe (isJust)
import Data.Word (byteSwap16)

oneNumTup :: Num a => (a, a)
oneNumTup = (2, 300)

doSomethingWith x y = byteSwap16 x > 10 && (y * 400) `div` 400 == y

main :: IO ()
main = print $ doSomethingWith (fst oneNumTup) (snd oneNumTup)

I'm beginning to think the only way to actually fix this will be to give hlint the ability to look at types.

There is one thing that all of my examples have in common: they rely in type defaulting. In my first and second examples, the rule removes code that had a type constraint, which made the type be ambiguous, and it then got defaulted to a different one. In my third example, the opposite happens: the rule adds an additional type constraint, so a type that used to be ambiguous and thus defaulted is now forced to be a new type.

The ability to look at types isn't feasible in the current version of HLint. In a future version, likely wired in as an extension to Ghcide, it becomes a lot more reasonable - but that's a long way off. Maybe we have to write that in the presence of type defaulting, code that might have done different things depending on how unspecified but inferred types operated might now do different things. I view such code as fairly fragile - I'd much rather we didn't break it, but the cost of not doing so seems high...

I added to the README:

Sometimes HLint will change the code in a way that causes values to default to different types, which may change the behaviour.

Definitely worth calling out.

Wait, I'm wrong again: by combining both of my approaches and applying two hints, I can make the problem occur without relying on the existence of type defaulting. Consider this program:
import Data.Maybe (isJust)
import Data.Word (byteSwap16)

oneNumTup :: Num a => (a, a)
oneNumTup = (2, 300)

doSomethingWith x y = if byteSwap16 x > 10 && (y * 400) `div` 400 == y then Just (x,y) else Nothing

bar = doSomethingWith (fst oneNumTup) (snd oneNumTup)

foo :: Num a => (a, Integer) -> a
foo (x,y) = x + fromInteger y

main :: IO ()
main = print $ isJust (fmap foo bar)

No type defaulting occurs either in it as-is, or in it with HLint's two hints, so even -Werror=type-defaults both before and after running HLint won't save you.

So the fundamental problem is that uncurry reduces the number of identifiers with the rule:
    - warn: {lhs: f (fst p) (snd p), rhs: uncurry f p}

So whereas the LHS has two instances of p, both of which were of type forall a . Num a => (a,a) - with different qualified a's, the RHS has only one instance of p and thus only one forall constraining two things to be the same type. I think this property is probably true for any hint which has two instances of a variable on the LHS and one on the RHS.
But those hints are often the most useful, and it's exactly the refactoring a human is likely to make by accident, so the code is pretty fragile in the first place. Maybe the solution is a well written article about how HLint can break the code, so we can cite it from the readme, so those people who really want to know can understand the precise problem and reasons?


Dependencies updated

Latest webpack, babel, css minifier…

:tada: This PR is included in version 3.0.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Adyen chargeback does not void the transaction

It happens quite frequently a transaction is chargebacked after a few seconds, see: http://i.imgur.com/JxmHI3d.png
But by then the order already has a paid invoice after having gotten the notification it was captured
and the invoice is not voided.
Basically it's not possible to determine if the invoice was paid or not.
I would expect the total due amount to be the order amount after a chargeback #

HI @sandermangel,
Because of the invoice already created the total due is zero.  Based on the CAPTURE notification you know if the payment was successfully paid.
Regards,
Rik
Adyen

Thanks for the answer, am I right to assume that any order with adyen_event_code "CAPTURE : TRUE" is successfully captured?
That would be a safe condition to forward to fulfillment?

Hi @sandermangel,
Yes indeed this is triggered by the notification and provide you the right information to sip the goods.
Regards,
Rik
Adyen


Reference currently supported connectors docs

The fuels-connectors documentation should maintain a list of all the currently supported connectors and their related wallet install links (issue). We should update this line to reference the list of connectors.

You can find a list of all the connectors in our FuelLabs/fuel-connectors.

Originally posted by @petertonysmith94 in https://github.com/FuelLabs/fuels-ts/pull/2786#discussion_r1688712378

Not planned on the connector side.

As it's not planned in the connectors, closing as not planned.


NameError: name 'piexif' is not defined

I set the correct parameters, but when I clicked on generate, it prompted that the 'piexif' module was not found?


Use asyncio-run-in-process to run trio components

Closes: #1435

I wasn't able to tell from scanning the patch: what was making the EOFError happen and how does this fix it?

I wasn't able to tell from scanning the patch: what was making the EOFError happen and how does this fix it?

We still don't know what was causing it, but it comes from the multiprocessing module, and this is the last place we were using that in trinity.

Huzzah,  the end of multiprocessing.


[Bug]: Salesforce OAuth token keeps expiring - Authenticated API datasource

Is there an existing issue for this?

[X] I have searched the existing issues

Description
https://theappsmith.slack.com/archives/CNQ9Q91C0/p1702972094026239
Steps To Reproduce
Follow the steps in this tutorial
https://community.appsmith.com/tutorial/how-integrate-salesforce-oauth2-appsmith
Public Sample App
No response
Environment
Production
Issue video log
No response
Version
v1.9.57-SNAPSHOT

The salesforce application configuration needs to be updated to uncheck the Require Secret for Web Server Flow and Require Secret for Refresh Token Flow checkboxes. Once this is done, redirection to salesforce login page will not happen and internally the token will be refreshed.


Add mongodb setup instructions

Making it a little more obvious how to setup/start the db.

:+1: Thanks dude!


Catboost bug

Catboost produces a TreeEnsemble has no "num_nodes" error with this code. Btw do you support a background dataset parameter, like in shap for "interventional" vs "tree_path_dependent"? Because if your underlying code uses the "interventional" method this might be related to this bug: https://github.com/slundberg/shap/issues/2557
from catboost import CatBoostRegressor
import  fasttreeshap

X, y = shap.datasets.boston()

model = CatBoostRegressor(task_type="CPU",logging_level="Silent").fit(X, y)
explainer = fasttreeshap.TreeExplainer(model, algorithm="v2", n_jobs=-1)
shap_values = explainer(X)

# visualize the first prediction's explanation
shap.plots.waterfall(shap_values[25])

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Input In [131], in <cell line: 13>()
     10 # explain the model's predictions using SHAP
     11 # (same syntax works for LightGBM, CatBoost, scikit-learn, transformers, Spark, etc.)
     12 explainer = fasttreeshap.TreeExplainer(model, algorithm="v2", n_jobs=-1)
---> 13 shap_values = explainer(X)
     15 # visualize the first prediction's explanation
     16 shap.plots.waterfall(shap_values[25])

File ~\miniconda3\envs\Master\lib\site-packages\fasttreeshap\explainers\_tree.py:256, in Tree.__call__(self, X, y, interactions, check_additivity)
    253     feature_names = getattr(self, "data_feature_names", None)
    255 if not interactions:
--> 256     v = self.shap_values(X, y=y, from_call=True, check_additivity=check_additivity, approximate=self.approximate)
    257 else:
    258     assert not self.approximate, "Approximate computation not yet supported for interaction effects!"

File ~\miniconda3\envs\Master\lib\site-packages\fasttreeshap\explainers\_tree.py:379, in Tree.shap_values(self, X, y, tree_limit, approximate, check_additivity, from_call)
    376 algorithm = self.algorithm
    377 if algorithm == "v2":
    378     # check if memory constraint is satisfied (check Section Notes in README.md for justifications of memory check conditions in function _memory_check)
--> 379     memory_check_1, memory_check_2 = self._memory_check(X)
    380     if memory_check_1:
    381         algorithm = "v2_1"

File ~\miniconda3\envs\Master\lib\site-packages\fasttreeshap\explainers\_tree.py:483, in Tree._memory_check(self, X)
    482 def _memory_check(self, X):
--> 483     max_leaves = (max(self.model.num_nodes) + 1) / 2
    484     max_combinations = 2**self.model.max_depth
    485     phi_dim = X.shape[0] * (X.shape[1] + 1) * self.model.num_outputs

AttributeError: 'TreeEnsemble' object has no attribute 'num_nodes'

Fixed this issue by avoiding "memory check" for CatBoost, since CatBoost is not supported in the current version of fasttreeshap (mentioned in https://github.com/linkedin/FastTreeSHAP/issues/6).
fasttreeshap is built only for "tree_path_dependent". You may still run "interventional" in fasttreeshap anyway, but its performance should be the same as in shap. I would suggest you to post issues related to "interventional" directly in shap GitHub page.

Is this also true For xgboost and lgbm? From my understanding in the Paper "tree path dependent" is the better Method For explaining model Performance and interventional is used to explain Relationships in the Data. Also "interventional" is a lot slower so wouldnt a fast tree shap Method make a lot of sense For it?

Yes. fasttreeshap accelerates the shap value computation for xgboost and lgbm only for "tree_path_dependent". It may make sense to accelerate "interventional" as well, however the algorithms used in "tree_path_dependent" and "interventional" are totally different. It is actually a lot harder to accelerate "interventional", and it is out of the scope of this package.


Introduce dry-run for hiveutil

Any plan to enable dry-run for hiveutil? This can help to facilitate the dev process.

Is this in relation to the create-cluster subcommand? If so does "-o yaml" meet your needs? That will just print the yaml we would send to the server, but nothing is actually sent.

@dgoodwin , not only the YAML templates, but I also want to know the work process for the hiveutil.
Just like helm install --dry-run at https://helm.sh/docs/helm/#synopsis-3 , it will only simulate the install process but does not do actual install.

any valuable install simulation would first require dry-run support in the OpenShift installer. I'm not aware of any support or any desire for such a thing.
Thank you for the suggestion, but unfortunately this is not a priority item for us.


slang-test/WGPU: unfilterable-float type sampler used to filter

This issue was split from https://github.com/shader-slang/slang/issues/5173.
Affected tests under tests/compute (currently excluded):

func-param-legalize

Output from slang-test, for func-param-legalize:
...
WGPU error: Texture binding (group:0, binding:0) is TextureSampleType::UnfilterableFloat but used statically with a sampler (group:0, binding:1) that's SamplerBindingType::Filtering
 - While validating compute stage ([ShaderModule (unlabeled)], entryPoint: computeMain).
 - While calling [Device].CreateComputePipeline([ComputePipelineDescriptor]).

WGPU error: [Invalid ComputePipeline (unlabeled)] is invalid.
 - While encoding [ComputePassEncoder (unlabeled)].SetPipeline([Invalid ComputePipeline (unlabeled)]).
 - While finishing [CommandEncoder (unlabeled)].

WGPU error: [Invalid CommandBuffer] is invalid.
 - While calling [Queue].Submit([[Invalid CommandBuffer]])
...

Generated WGSL for slangc -o test.wgsl -target wgsl -stage compute -entry computeMain %SLANG_SRC%\tests\compute\func-param-legalize.slang:
@binding(0) @group(0) var diffuseMap_0 : texture_2d<f32>;

@binding(1) @group(0) var samplerState_0 : sampler;

@binding(2) @group(0) var<storage, read_write> outputBuffer_0 : array<f32>;

struct Param_0
{
     base_0 : f32,
};

fn run_0( p_0 : Param_0,  p_tex_0 : texture_2d<f32>,  p_samplerState_0 : sampler) -> vec4<f32>
{
    return (textureSampleLevel((p_tex_0), (p_samplerState_0), (vec2<f32>(0.0f)), (0.0f))) + p_0.base_0;
}

@compute
@workgroup_size(1, 1, 1)
fn computeMain(@builtin(global_invocation_id) dispatchThreadID_0 : vec3<u32>)
{
    var p_1 : Param_0;
    p_1.base_0 = -0.5f;
    var _S1 : vec4<f32> = run_0(p_1, diffuseMap_0, samplerState_0);
    outputBuffer_0[i32(0)] = _S1.x;
    outputBuffer_0[i32(1)] = _S1.y;
    outputBuffer_0[i32(2)] = _S1.z;
    outputBuffer_0[i32(3)] = _S1.w;
    return;
}

Yeah it seems we just return unfilterable-float unconditionally.
https://github.com/shader-slang/slang-rhi/blob/84f8ce13f92e9538b2e15f03d6bdac5ad9af215a/src/wgpu/wgpu-shader-object-layout.cpp#L54

As far as I know, textures are not available on compute stage with WGSL. It works only for the fragment shader.

I would be super surprised if textures are not available in compute stage ...

...is TextureSampleType::UnfilterableFloat but used statically with a sampler (group:0, binding:1) that's SamplerBindingType::Filtering

Oh, I think possibly the fix here is to make the sampler not be of type filtering -- I don't know where it would get derivatives and such generally used for filtering, in a compute shader.

I was surprised too but may be I am reading it wrong.
Please see it for your self, here

16.7.8. textureSample
Samples a texture.
Must only be used in a fragment shader stage.

I was surprised too but may be I am reading it wrong. Please see it for your self, here

16.7.8. textureSample
Samples a texture.
Must only be used in a fragment shader stage.

It could also be because you need derivative of screen-to-triangle transformation in order to pick MIP level.
https://www.w3.org/TR/WGSL/#texturesamplelevel seems to be fine. (Check compute.toys samples)

Oh. Yes, that must be it.
It makes sense now


there is a display bug with chinese

Cannot show Chinese language correctly。
url : https://github-history.netlify.com/llyer/leetcode/blob/master/README.md

It's not only Chinese: it's unicode characters in general.
I use non-breakable spaces in my code sometimes, and they don't show up properly either:

(tested here)

It's not only Chinese: it's unicode characters in general.
I use non-breakable spaces in my code sometimes, and they don't show up properly either:

(tested here)

Ok, understand.
Will you fix this problem?

I'm not any maintainer of this project, I just added more details to the issue (= not only chinese but all unicode chars), and I don't know how the project is coded 😄

Thanks, I think I know what's the problem, I'll fixit soon

#51 should fix this


As an author, I want un-befriend local and remote authors

@amalik2  don't you already have this done? https://github.com/i-5ive/cmput404-project/blob/master/react/src/js/react/profile/ProfileStore.js#L253

@MandyMeindersma This isn't implemented for remote authors, so it should not be closed

Yea


Laravel 5.8 support

Any plan to support Laravel 5.8?

Bump

Bump

FYI: I've opened #35 which bumps dependencies to Laravel 5.8

Can't it get merged?

Merged #35
Can be closed


AnayaDesign tracks steps of GitHub Flow

[x] branch off of master
[x] create commits on our new branch
[x] open a PR on GitHub
[x] code review + collaboration
[x] merge when we receive 👍

function ohSnap(whatsUp) {
  return whatsUp + " are awesome"
}

done done. 💯


sirf_registration notebook example

1.) When run in Jupyter from Docker, uses Python 2 and hence the print statements are ugly (though readable).
2.) Clarity: Does the 1 in this statement get_dimensions()[1] refer to the first dimension? I thought python labeled the first dimension as 0? Oh, and python throws in another bit of confusion by putting the slice in this dimension.
My point is that to anyone not versed in the quirks of python, this is quite hard to follow and needs some comments in the example.
3.) Does GeometricalInfo have a print method? This could be handy.
4.) The example might be made clearer by splitting into a few examples. First, two images that look similar apart from an in-plane displacement and then registering them. Second, two images that look like those in the example but with no real-world displacement (to show that the registration can take into account the spatial referencing provided by the geometrical info - the expected displacement field would be the identity). Third the full thing, but state at the outset how these images differ in their real-world position (at the moment it's not very clear if they are a fixed head with incorrect geometrical info, or there has been some movement between them that we are trying to find by registration).

Sorry not sure how to help on that one. Is it possible to use python3?

Since it's a NifttImageData, the nifti convention is used here, in which the dimensions contain 7 values: [ndim, nx, ny, nz, nt, nu, nv, nw]. So here we used get_dimensions()[1] to get the number of voxels in the x direction. You're right that none of this is explained in the notebook and should probably be added. I suppose I omitted this in an attempt to not overcomplicating things.

I just had a look and it seems that GeometricalInfo has a method called print_info. It would be trivial to add a __str__ method, which means you could do print(ref.get_geometrical_info()). I'm not sure why we didn't do this already.

P.S.: Confusingly, the GeometricalInfo would use [nx,ny,nz] which would mean that the 0th index would correspond to the 1st index of NifttImageData.get_dimensions(). I think it is somewhat unavoidable that the individually wrapped functionality (e.g., NiftiImageData.get_dimensions) are slightly different to native SIRF functionality (e.g., GeometricalInfo) since each package has its own way of ordering dimensions. STIR for example has the z dimension first.

@DANAJK is this issue relevant/fixable for 3.1?

I have made an alternative notebook that is on the Teams site.

all done I believe.