Make R languageserver installation respect alternative/user-defined options("repos")

@williamboman - hope this is the right place to address this?
Problem description - "steps to reproduce" (but only if you have a somewhat strict firewall in place, if there is not tightened security around, this will not be an issue, prob just a feature?).
Start Neovim
LspInstall (or MasonInstall) is expected to be available.
Not mandatory: open a .R file
Hit ":" (to enter Command mode) type LspInstall r_language_server (or MasonInstall)
Watch it use "http://cran.rstudio.com/" which in my case fails setting up the lang server for neovim b/c some packages (like callr, rprojroot, etc.) will be blocked from downloading from this mirror but not an "internal" one defined in my ~/.Rprofile.
Found that to be caused by line 8 in /src/install.r
options(repos = list(CRAN = "http://cran.rstudio.com/"))

which overwrites my personal [CRAN] mirror.
Tbh as mentioned above I don't know if this more of a feature request than a "real" issue?
I tried the "easy" or some form of workaround ie forking this repo, changed that line 8 in install.r and adjust the masons extensions registry.json file line xxx to point to that personal fork (during the install routine, sort of), but any changes to that registry.json file will get overwritten prior to/during the LspInstall process :-(
And I am not familiar enough with the mason or lsp extensions inner workings to dig deeper into further workarounds or changes to their lsp installation processes.
Therefore it would be really helpful if there was a way to avoid forcing that 'cran.rstudio.com' repo into the package installation process.
Alternatively, maybe there's a setting somewhere to use an already installed instance of library(languageserver), instead of triggering a full "new install" of the languageserver package, which I just have not been able to find (yet) somewhere eg in the documentation?
Also no problem to go down that latter route - if someone can point me in the right direction?

Hi - not a maintainer of this repo, but I have been working on similarly integrating R with Neovim and have a few brief suggestions.
Check that you've added GPG keys for CRAN repos in your distro. See [this guide] (https://blog.zenggyu.com/posts/en/2018-01-29-installing-r-r-packages-e-g-tidyverse-and-rstudio-on-ubuntu-linux/index.html) to ensure R is in the best compatible state for installing packages.
This fixed my own problems getting R up and running in Linux. I'm not entirely certain what magic happens here, but I think the core problem is installing packages that are out of sync with the current version of R, which causes issues when installing r-languageserver.
Second, make sure you R_PROFILE_USER environment variable is set - see this . Otherwise, it will default to the .Rprofile in the home directory (~) rather than your working environment.
In short, although you are using a fork of line 8, it is still loading a .Rprofile from ~. If no .Rprofile is found at ~, it is defaulting to the Rstudio CRAN repository. IIRC because Mason using lazy installing, the reinstall process will use the main fork, which resets the modifications you've made by using the main fork.
Neovim support for R and the R-LSP is still very underdeveloped which will hopefully change in the next few years. Alternatively, ensuring you point directly to local repositories of precompiled-binaries may dramatically decrease the length of time with the Mason/LspInstall commands. Hope this helps a bit!

@harmongray Hi thanks for the suggestions, I will give some of them a shot.
GPG Keys and similar things are not an issue (in my case) and anyway could not or cannot be modified or managed by me on the infrastructure I am working with here.
It is "only" certain types of packages that fail the download from "outside sources".
The firewall in essence prevents code that would try to build after being downloaded to pass through (so my guess is that anything that has a Makefile in it somewhere gets blocked, though I do not know the full details).
That is what makes for example callr (despite afaict only pure R code in it) fail building  (after downloading) b/c callr depends on processx which will not get or was not installed in the first place b/c that has Makefiles around and therefore is not available - the story continues like that :-(
All of this can only be remedied if the installation would honor the ~/.Rprofile settings, which define a "safe" internal CRAN for this kind of actions.
If the MasonInstall would do such a thing as default to my home directory ~/.Rprofile that would be great, since in that case I wouldn't have had to open this ticket in the first place :-)
Also tried exporting the R_PROFILE_USER environment variable you mentioned and eg reducing my .Rprofile to simple one liner  ie options(repos='my.internal.mirror') but to no avail :-(
I also think that the blob you are linking to has nothing to do with the code in this mason-org r-languageserver repo? At least I was not able to find any routine in any file in here, that would check on any "alternative" sources for using different CRAN mirrors then the one hard-coded in line 8 of /src/install.r.
Still thanks a lot for the comments and suggestions.
So for now I guess I'm gonna continue w/o a r-languageserver in NeoVim and hope that the maintainer will stumble across this at some later time.

Just for clarity on my initial suggestion, the end of the code features this section:
# We set force = TRUE because this command will error if languageserversetup is
# already installed (even if it's at a different library location).
remotes::install_github("jozefhajnala/languageserversetup", lib = rls_lib, force = TRUE)
if (did_install_remotes) {
  remove.packages("remotes", lib = rls_lib)
}

loadNamespace("languageserversetup", lib.loc = rls_lib)
languageserversetup::languageserver_install(
  fullReinstall = FALSE,
  confirmBeforeInstall = FALSE,
  strictLibrary = TRUE,
  ref = ref
)
library("languageserver", lib.loc = rls_lib)

which resets the .Rprofile via a remote GitHub installation of "languageserversetup," which then ensures the library is installed in the separate location. I haven't entirely tested this entirely but I'll be fiddling the next few days to avoid compiling during Mason Install time. My other goal will be to securely set the .RProfile without overrides.
If you can get the work around working, you can load preconfigured binaries into Mason or LspInstall, solving both the load time and the firewall issue. I am also wrestling to get nvim working correctly, so I understand the frustration. Best of luck in your endeavors!

@harmongray  thnx for pinpointing me to that ... totally skipped my eye that there's another languageserver thingy around in that file

@harmongray tinkering around basaed on that other package does not change much for me ... :-(

You are correct that the modification of R_PROFILE_USER was not the root cause of this problem. However, its still good practice I recommend setting it in the future as it should be the source of a workaround sometime in the future.
The issue comes down to the way that Mason handles its registries. Mason recently switched to a new registry mechanism. While it also allows pinned registries for historical version control, it is still not very compatible with intra-net without significant hacky workarounds.
Temporarily modifying your does not work because Mason immediately requests an update to the registry. This depends on a variety of verification/checksum mechanisms that cannot fail. There are several new-ish issues that have been opened (#1300, #1341, and #1526) to discuss how to get around this behavior with mostly negative results.
While Mason only recently implemented pinned registries, the most likely solution to this in long term would be permitting the configuration of an "unsafe" registry method to allow for intranet and "offline" users. This is does not appear to be anywhere on the horizon.
In short, the root of the problem has been identified by other users and there is a small amount of interest in this feature, it might happens somewhere down the line. In the mean time, this is a fun pet project and I'll be looking into what it might take to implement it.
In the mean time, a non-Mason use of r-languageserver may allow you to modify this code. The repository page we are on currently is a custom wrapper for r-languageserver shows support with a vim Plug  and Coc, which also manages LSPs.

I just moved away from Coc a couple of weeks ago in favor of Lsp and Mason and tbh I very much prefer neovim lsp for basically all my other LSP's in place (py, julia, rust, ...) I am just missing the R languageserver behind my firewall.
I don't think it is as much a Mason registry mechanism as simply the install.R file line 8 not forcing a hard coded CRAN mirror URL into the install process there. Dunno ...

Two more troubleshooting questions:
Can you post the output of a tcpdump or similar tool to validate that it is a firewall issue, as well as the output of the most recent MasonLog?
Have you ensured that R is installed properly with the method I mentioned earlier? This is important because R often does not allow for out of date packages. In this case, r-languageserver originally did not compile for me until I reinstalled. This is the issue I had with setting it up initially.
The default R repository for apt has often been out of sync with versions, causing these exact problems.
While it still may be a firewall issue, a later fix for this specific Mason method may be possible with strategic use of the bspm within install.packages() within the langugeserversetup, which could eliminate these compile times.

It's as simple as "I can install anything from the local mirror, defined in my .RProfile" as soon as I change that mirror to any other address that is not this internal mirror any packages that contains a Makefile will be blocked from being downloaded (even from official CRAN mirror[s]).
So investigating into tcpdumps will not change anything for me - which anyway I cannot get my hands on, sine I am not managing the proxy.
The solution to my problem will still remain to 'use the mirror defined in .RProfile'.
Tweaking anything in Mason, will also not do me any good if the line 8 in install.r here does not change.

Hello! I'd like to preface this by saying that I'm not a R developer so my understanding is very limited. Would removing that line altogether be a good solution? I can't recall why it's included, and I'd like to believe that without it R would simply use the default repository (or whatever is configured in the user's profile).

@williamboman

Hello! I'd like to preface this by saying that I'm not a R developer so my understanding is very limited. Would removing that line altogether be a good solution? I can't recall why it's included, and I'd like to believe that without it R would simply use the default repository (or whatever is configured in the user's profile).

Still "spot on" despite not being a R developer ;- )
... that is exactly what would/should happen if that line is commented out/removed.
The line you currently have in there basically takes precedence above any settings that would take effect, because the files in the bin/ folder invoke R with -f (ie file to execute parameter) followed by install.r and settings options in that file has fairly huge effetcts in the way that a lot of function calls in R have options defined this way as default theri values for various function calls
eg.  for install.packages the first few args are:
function (pkgs, lib, repos = getOption("repos"), ... # getOption gets the value defined via options("repos" = list(... in install.r

And judging from the files in the bin/ folder (ie install.ps1, install, etc.) there are also no calls of the form --no-init or similar that would force ignoring the users .Rprofile or so in place - so I would stand with my analysis, that this should solve that issue.
Greetings,
W

The code in Line 8 should be modified to check for the existence of the .RProfile or global options and inheriting the CRAN repository in those locations.
Otherwise, it may break the functionality within Mason, as it will ask which cran repository to use if not set in global options. If not set, this often triggers a GUI/list to choose from. Checking for an existing CRAN repo allows user defined behavior while ensuring an option is set so that dependencies may be properly installed.

I often write in R and can make a PR to check for this sometime in the next few days so that this use case is addressed and the core functionality of Mason is unimpeded.

Great! Apologies for the late response btw, been pretty busy lately. I'll get it merged later today and rerelease the latest version

Reference

Guess you are totally right that some form of check
if (getOption("repos") == "@CRAN@")  options("repos" = list(....)) is the smarter way to go here.

I've opened https://github.com/mason-org/r-languageserver/pull/3 now, wdyt?


[.gitignore] Ignore dist/ dir

+1 for this !

close with no interest by people.

Not sure what the general consensus is: I personnally ignore dist directories in my global gitignore, but that may be a bad idea...


Update panel in a single TextCommand

Prevents panel from flickering due to multiple commands updatating its
contents (fixes #700).
Also try to restore the previous selection to allow the user to continue
browsing errors exactly were they left off. Reset to the beginning
otherwise.

This would replace #704 then I presume?

Haha, looks like I wasn't fast enough.

the presumption was pretty strong ;) Thanks for this!


Bug in dropdowns with ajax

Hi, I've found a bug that makes dropdowns non-functional with ajax updates. If you place an ink dropdown somewhere in a page, it works, but when a section of a page containing a dropdown gets updated with ajax, the dropdown functionality doesn't work anymore.
Hope this issue gets resolved, I like Ink much better than bootstrap, but this is a serious problem in my app.
Thanks!

Hello there again, just wondering if you have found some kind of solution to this problem? I've grown very fond of InK and use it on multiple project, even brought it to my company, but this stays to be a bit of a problem, since the workaround is not the cleaniest peace of code :-)
Thanks a lot, Johny


add crp4 configuration

For reference to CRP4 test: https://indico.fnal.gov/event/64497/contributions/290688/attachments/177546/241768/240503_CRP45_reconstruction.pdf


When two users are on the same conversation they swap users.

Summary
Hi, I have found a bug that has been happening a long time for me but I hoped it would be fixed so now I open an Issue to let the community know about it. The bug is related when two users are in the same conversation one of them receives a popup of "lost internet connection". After that, if they refresh the page they are converted to the other user that was watching the conversation too. After that, if now with the converted user, you log out and enter with yours, when you log out you also kick the other person out of their user.
Steps to reproduce
It simply happens when two people are in a conversation and you can test scrolling or changing status to try to force it even faster.
What is the expected correct behavior?

Relevant logs and/or screenshots

Environment
Client: Docker Engine - Community
Version:           25.0.4
API version:       1.44
Go version:        go1.21.8
Git commit:        1a576c5
Built:             Wed Mar  6 16:32:12 2024
OS/Arch:           linux/amd64
Context:           default
Server: Docker Engine - Community
Engine:
Version:          25.0.4
API version:      1.44 (minimum version 1.24)
Go version:       go1.21.8
Git commit:       061aa95
Built:            Wed Mar  6 16:32:12 2024
OS/Arch:          linux/amd64
Experimental:     false
containerd:
Version:          1.6.28
GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
runc:
Version:          1.1.12
GitCommit:        v1.1.12-0-g51d5e94
docker-init:
Version:          0.19.0
GitCommit:        de40ad0

Image version / tag: tiredofit/freescout:latest
Host OS: Ubuntu 22.04

Are you running NGINX Proxy Manager by chance?

@alexskrms are you running NGINX Proxy Manager by chance?

Yes @sparker81

You will need to turn off the Cache Assets setting if you have it turned on for Freescout in NGINX Proxy Manager. I had to find that out the hard way. https://github.com/freescout-helpdesk/freescout/issues/3837

Here's a screenshot. Hopefully it fixes the issue.

Here's a screenshot. Hopefully it fixes the issue.

Thanks you soo much solved!


Support image names

Right now it seems to only get container names, but it would be nice to get image names for docker run, docker rmi, and any other command that works with images.
Right now, docker run will show container names if you tab, which is wrong.

This is indeed something I would like to add in the future.

This is now implemented in 0.5.1.
Let me know if it works for you! Run Update-Module posh-docker and restart your console to see if it works.


Add: GitHub Readme Quotes under Tools

What type of PR is this? (check all applicable)

[x] 🚀 Added Name
[ ] ✨ Feature
[ ] ✅ Joined Community
[ ] 🌟 ed the repo
[ ] 🐛 Grammatical Error
[x] 📝 Documentation Update
[ ] 🚩 Other

Description
Added GitHub Readme Quotes under tools.
Add Link of GitHub Profile
Github

@all-contributors please add PiyushSuthar for doc


Bump cardano-node to pioneer-wave2-38-g1c797366

Latest master rev 1c79736 fixes an issue where the cardano-node sources can't be checked out on a case-insensitive filesystem (macOS) ⇒ input-output-hk/cardano-node#1131.

Byron integration tests passed but shelley integration tests failed to finish... Will update the base libraries too.

bors r+

bors r-

bors try

@Anviking Do you think we can merge this and trigger the release, or trigger it first?

No need for the cardano-haskell snapshot though, since this is still an intermediate release.
Let's stick with local snapshots until things stabilize.

— @KtorZ
bors r+


Angular2 directives not working in components templates?

Hello everyone,
I am working with Angular2 Project.
What I wanted to do is to add simple ngFor loop inside any template, but it seems to fail - It shows nothnig.. also there is no navigation panel. Looks like whole page is broken.
My dashboard.component.html:

            {{item}}

My dashboard.component.ts:

import { Component, OnInit }    from '@angular/core';

@Component({
    templateUrl: 'dashboard.component.html',
})
export class DashboardComponent {

    items: Array<any> = [
                { id: 1, text: 'Option #1' },
                { id: 2, text: 'Option #2' },
                { id: 3, text: 'Option #3' },
            ];

    constructor() {
    }
}

dashboard.module.ts:
import { NgModule }                 from '@angular/core';
import { ChartsModule }             from 'ng2-charts/ng2-charts';

import { DashboardComponent }       from './dashboard.component';
import { DashboardRoutingModule }   from './dashboard-routing.module';
import { SelectModule }             from 'ng2-select'
import { Ng2BootstrapModule }       from 'ng2-bootstrap/ng2-bootstrap';

@NgModule({
    imports: [
        DashboardRoutingModule,
        ChartsModule,
        SelectModule,
        Ng2BootstrapModule
    ],
    declarations: [ DashboardComponent ],
})
export class DashboardModule { }

Does anybody have same problems? Or am I missing something?

It look like there are missing some imports. Found Solution here: http://stackoverflow.com/questions/34228971/cant-bind-to-ng-forof-since-it-isnt-a-known-native-property
CommonModule need to be imported in for all children modules.


New package: ModiaPlot_WGLMakie v0.1.0

Registering package: ModiaPlot_WGLMakie
Repository: https://github.com/ModiaSim/ModiaPlot_WGLMakie.jl
Created by: @MartinOtter
Version: v0.1.0
Commit: 420b330c8c04b5a525246bc1533dff6b2cf397e7
Git reference: main
Release notes:

First public release (based on ModiaResult 0.2.0)

Please, cancel this pull request (this version missed a "pull". Had started a new registration)


Error while compiling the singularity container

Hi,
I am having trouble following the said instructions, where I get the following error:
/.post.script: line 123: cd: /opt/conda/lib/python3.7/site-packages/colabfold/: No such file or directory
/.post.script: line 150: cd: /opt/conda/lib/python3.7/site-packages/colabfold/: No such file or directory
FATAL:   While performing build: while running engine: exit status 1
I am trying to compile this for cluster use and am not super familiar with colabfold. Which brings me to my 2 questions I have:
If I were to change your singularity with a locally installed non container colabfold version, where do I make the appropriate changes in your code.
Or if the singularity is really required, please can you help me solve the aforementioned issue.
Lastly, I am trying to install this on Debian12 but I don't believe that should be the issue as the container should be os independent

Hello, thanks for your interest.
Are you running the singularity build command with root privileges? As said in the instructions "To create the colabfold singularity image, root privilege is needed and the command below may be more easily performed on a local linux machine even without GPUs. Once built, the file colabfold_container_v13.sif can be copied to the machine running the scan_idr pipeline such as a server or a cluster preferentially with GPUs."
Otherwise, if you would like to run ColabFold independently of the container version, you would need to change the run_colab_template.sh script (where we are running colabfold_batch through the singularity container).
Please let us know if this solves your problem, otherwise we'll investigate possible workarounds.

Yes I am running the installation of the singularity container as sudo on my local machine but I can just use the second hint you provided to go around this issue as I have a working colabfold version available. Thank you

Hello, did you solve this? Thank you

Hi, I was installing this on our HPC environment for a user, I believe I got around it with the workaround that the user will have a copy of the pipeline, which they can edit to their needs (as we have currently multiple colabfold versions and 2 databases) and run it. However, they have not tested it as of now. Please feel free to close this as solved for now and in case we run across issues I will get back to you.
Also I did this in a python virtual environment with all the other non-python conda dependencies being loaded through a module system.

Thanks for your feedback! As you suggested, closing the issue for now, please let us know if you have other issues.


Mythic Ability:  Unlock the equipment bar of Animal Companions

Baphomet's skill can  Unlock the equipment bar of Animal Companions ,why not use a Mythic Ability to unlock the equipment bar of animal companions to give them more power？

I can't find that feature. Do you have the guid or exact name?

In the old version, when Pavlovine used the ability:Baphomet_BaphometDisjunction(GUID:24511c87d62447ae80c57336a9c212a4) against animal companions.
This will clear all buffs and unlock the  animal companions's equipment bar.I think it's a bug, but it's interesting.

And if remove AnimalCompanionSlotFeature（GUID:75bb2b3c41c99e041b4743fdb16a4289），Animal Companion can use equipment bar.

thx your work, your mod is really good.I won't open the game without your mod


issues

解释.txt

解释.txt

Python3就会这样，这里的版本应该是Python2

解释.txt

https://github.com/luohaifenglight/snowflakeservice.git   我这个支持python2,3 并且无依赖，可以自己方便改造。也可以独立使用做成分布式服务。

Not sure why you have opened an issue. I see that you have taken the idworker.py and put it in a new repo and slightly changed it so it won't have any dependencies on Thrift.


The cuda problem is not being solved

CUDA backend failed to initialize: Found cuDNN version 8700, but JAX was built against version 8904, which is newer. The copy of cuDNN that is installed must be at least as new as the version against which JAX was built. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
Nothing is working with this error. How can I solve this problem?

I ran TheLastBens sd auto1111 colab and also since Friday I get the following error message:
WARNING[XFORMERS]:
xFormers can't load C++/CUDA extensions. xFormers was built for:
PyTorch 2.1.0+cu118 with CUDA 1106 (you have 2.1.0+cu121)
Python 3.9.16 (you have 3.10.12) Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
Memory-efficient attention, SwiGLU, sparse and more won't be available.
Set XFORMERS_MORE_DETAILS=1 for more details
Is it within Colab/ notebook version issues or is it in someway connected to my local versions? Because which python --version says I'm running on 3.12.

Yeah, as I said above almost every colab related to SD is broken due to Google latest update. There is a temporary fix, I don't know if it will work for Kohya. Check out my topic on The Last Ben's discussion group.

Good to know — thank you! What’s/ where is TheLastBens Discussion group?

The cuda fix is pushed, it may still showing cuda is not initialized or something but I finished a LoRA training without error, the problem actually in bitsandbytes version.

The cuda fix is pushed, it may still showing cuda is not initialized or something but I finished a LoRA training without error, the problem actually in bitsandbytes version.

It can works, thanks for your efforts.
Your trainer has the best training quality among other Colab notes.

I'm able to train using DAdaption but not Adam8bit as a result of this.


Second attempt at a rust implementation

Is it possible to decouple the Golang in-file changes from this? It's a huge diff already and they make it a bit harder to review.

I'd rather not - the only Golang changes should be moving files and replacing file watching with SIGHUP. I can maybe look into porting some of the changes into the go PR and removing them from here, but I'd like to avoid additional work if possible.


Errors building Mantis from source following official docs

Summary
Hey folks. I'm having some difficulty building the complete Mantis project, as well as its examples from source.
I assume this is my environment being incorrectly configured rather than an issue with the master branch.
It would be awesome if we could figure out what's missing and update the docs to include those dependencies.
Thanks for taking a look!
Reproduction
I get errors when following the build instructions in the README:
$ git clone git@github.com:Netflix/mantis.git
$ cd mantis
$ ./gradlew clean build
BUILD FAILED in 4s

I get similar errors when following the instructions from the "Local" tutorial:
$ git clone git@github.com:Netflix/mantis.git
$ cd mantis/mantis-examples/mantis-examples-synthetic-sourcejob
$ ../../gradlew execute
BUILD FAILED in 3s

I've demonstrated my environment is at least somewhat correctly setup by successfully building mantis-mql from source:
$ git clone https://github.com/Netflix/mantis-mql
$ ./gradlew clean build
BUILD SUCCESSFUL in 52s

Here's some metadata around what I believe is being used to build Mantis:
$ ./gradlew --version

------------------------------------------------------------
Gradle 7.5
------------------------------------------------------------

Build time:   2022-07-14 12:48:15 UTC
Revision:     c7db7b958189ad2b0c1472b6fe663e6d654a5103

Kotlin:       1.6.21
Groovy:       3.0.10
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          17.0.6 (Eclipse Adoptium 17.0.6+10)
OS:           Mac OS X 14.1.2 aarch64

Details
It looks like at least two tasks are failing with quite loud stacktraces which I'm partially summarizing below:
> Task :mantis-common-serde:spotlessJava FAILED
Step 'removeUnusedImports' found problem in 'mantis-common-serde/src/main/java/io/mantisrx/common/JsonSerializer.java':
null
java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.IllegalAccessError: class com.google.googlejavaformat.java.RemoveUnusedImports (in unnamed module @0x2c6b2f25) cannot access class com.sun.tools.javac.util.Context (in module jdk.compiler) because module jdk.compiler does not export com.sun.tools.javac.util to unnamed module @0x2c6b2f25
        at com.google.googlejavaformat.java.RemoveUnusedImports.removeUnusedImports(RemoveUnusedImports.java:188)
        ... 136 more

> Task :mantis-common:compileJava FAILED
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OperatorOnErrorResumeNextViaFunction.java:76: warning: [deprecation] getInstance() in RxJavaPlugins has been deprecated
                    RxJavaPlugins.getInstance().getErrorHandler().handleError(e);
                                 ^
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OperatorOnErrorResumeNextViaFunction.java:76: warning: [deprecation] handleError(Throwable) in RxJavaErrorHandler has been deprecated
                    RxJavaPlugins.getInstance().getErrorHandler().handleError(e);
                                                                 ^
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OnSubscribeRedo.java:79: warning: [deprecation] <T>create(OnSubscribe<T>) in Observable has been deprecated
        return create(new OnSubscribeRedo<T>(source, notificationHandler, true, false, Schedulers.trampoline()));
               ^
  where T is a type-variable:
    T extends Object declared in method <T>create(OnSubscribe<T>)
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OnSubscribeRedo.java:83: warning: [deprecation] <T>create(OnSubscribe<T>) in Observable has been deprecated
        return create(new OnSubscribeRedo<T>(source, notificationHandler, true, false, scheduler));
               ^
  where T is a type-variable:
    T extends Object declared in method <T>create(OnSubscribe<T>)
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OnSubscribeRedo.java:108: warning: [deprecation] <T>create(OnSubscribe<T>) in Observable has been deprecated
        return create(new OnSubscribeRedo<T>(source, notificationHandler, false, true, Schedulers.trampoline()));
               ^
  where T is a type-variable:
    T extends Object declared in method <T>create(OnSubscribe<T>)
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OnSubscribeRedo.java:112: warning: [deprecation] <T>create(OnSubscribe<T>) in Observable has been deprecated
        return create(new OnSubscribeRedo<T>(source, notificationHandler, false, true, scheduler));
               ^
  where T is a type-variable:
    T extends Object declared in method <T>create(OnSubscribe<T>)
/Users/jpittis/mantis/mantis-common/src/main/java/io/reactivx/mantis/operators/OnSubscribeRedo.java:116: warning: [deprecation] <T>create(OnSubscribe<T>) in Observable has been deprecated
        return create(new OnSubscribeRedo<T>(source, notificationHandler, false, false, scheduler));
               ^
  where T is a type-variable:
    T extends Object declared in method <T>create(OnSubscribe<T>)
7 warnings
An exception has occurred in the compiler (17.0.6). Please file a bug against the Java compiler via the Java bug reporting page (https://bugreport.java.com) after checking the Bug Database (https://bugs.java.com) for duplicates. Include your program, the following diagnostic, and the parameters passed to the Java compiler in your report. Thank you.
java.lang.NullPointerException: Cannot read field "bindingsWhenTrue" because "currentBindings" is null
        at jdk.compiler/com.sun.tools.javac.comp.Attr.handleSwitch(Attr.java:1794)
...

I think this issue is due to JDk17. Try using JDK8 to build. https://sdkman.io/ would be helpful in managing multiple java versions.

Thanks so much. After installing sdkman, I was able to successfully build Mantis using JDK8:
$ sdk install java 8.0.392-amzn
$ ./gradlew --info --stacktrace build --warning-mode=all
BUILD SUCCESSFUL in 4m 6s

We should action this ticket by updating the docs from claiming "JDK 8 or higher" to just "JDK 8" and potentially mention sdkman as well as a recommended vendor and version.


Unable to access postgres-postgis with .env file values

Description:
Running docker-compose up -d postgres-postgis does not create a database with the variables set in the .env file (DB: default, user: default, password: secret). Attempting to access the DB using any role (root, postgres, etc) fails saying the role does not exist.
Manually specifying the role and changing the data path does work, e.g.
### PostgreSQL PostGis ###################################
    postgres-postgis:
      build: ./postgres-postgis
      volumes:
        - ${DATA_PATH_HOST}/postgres:/var/lib/postgresql-11/data # Note change of path here
      ports:
        - "${POSTGRES_PORT}:5432"
      environment:
        - POSTGRES_DB=default # Manually specified
        - POSTGRES_USER=postgres # Manually specified
        - POSTGRES_PASSWORD=postgres # Manually specified
      networks:
        - backend

Connecting to <IP_ADDRESS> using postgres/postgres shows the databases correctly and allows psql to work through bash.
Expected Behavior:
Creation of role "default" with password "secret" for access to database "default", as specified in .env.
Context information:
Output of git rev-parse HEAD
05a45ed0c86f30cce339f1b693a07d6cd169aa7f

Output of docker version
Client: Docker Engine - Community
 Version:           20.10.1
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        831ebea
 Built:             Tue Dec 15 04:34:59 2020
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       f001486
  Built:            Tue Dec 15 04:32:40 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker-compose version
docker-compose version 1.25.0, build unknown
docker-py version: 4.1.0
CPython version: 3.8.5
OpenSSL version: OpenSSL 1.1.1i  8 Dec 2020

System info: Mac, Windows or Linux. Include which disto/version
20.04.2 LTS (Focal Fossa)

Steps to reproduce the issue:

docker-compose up -d postgres-postgis with .env vars set to defaults
Attempt to connect to <IP_ADDRESS> with any PG client --> FAILS
Set vars manual and rebuild
Attempt to connect to <IP_ADDRESS> with any PG client --> Succeeds

try upgrade docker-compose

I am having this exact issue in
Ubuntu 20.04.2 LTS (Focal Fossa)
▶ docker-compose up postgres-postgis
Starting laradock_postgres-postgis_1 ... done
Attaching to laradock_postgres-postgis_1
postgres-postgis_1     | 2021-06-03 01:17:40.596 UTC [1] FATAL:  database files are incompatible with server
postgres-postgis_1     | 2021-06-03 01:17:40.596 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 13, which is not compatible with this version 11.2 (Debian 11.2-1.pgdg90+1).
laradock_postgres-postgis_1 exited with code 1

@alexc-tti  Did you resolve this issue?
Here is the problem and solution.
Problem: It was using mdillon/postgis which was not updated for 2 years.
FROM mdillon/postgis:latest

LABEL maintainer="Mahmoud Zalt<EMAIL_ADDRESS>
CMD ["postgres"]

EXPOSE 5432

Solution is change FROM to  postgis/postgis:latest
laradock/postgres-postgis/Dokerfile
FROM postgis/postgis:latest

LABEL maintainer="Mahmoud Zalt<EMAIL_ADDRESS>
CMD ["postgres"]

EXPOSE 5432

But changing this file is not enough you have to fully upgrade laradock to latest version
Then you either have to rebuild or do a full reset and install everything from scratch.
Now in your .env file in laravel project.
DB_CONNECTION=pgsql
DB_HOST=postgres-postgis
DB_PORT=5432
DB_DATABASE=astrareal
DB_USERNAME=default
DB_PASSWORD=secret

@bestlong You can probably close this issue now.

#2925


⚠ Bofrost.at has degraded performance

In 73e8770, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1548 ms

Resolved: Bofrost.at performance has improved in d84dbed.


Knowledge Graph is empty

Describe the bug
When I upload file and try to create kg. After processing the file in container logs following error appers:
2024-08-27T09:10:36.447361443Z  WARNING:neo4j.notifications:Received notification from DBMS server: {severity: WARNING} {code: Neo.ClientNotification.Statement.UnknownPropertyKeyWarning} {category: UNRECOGNIZED} {title: The provided property key is not in the database} {description: One of the property names in your query is not available in the database, make sure you didn't misspell it or that the label is available when you run this statement in your application (the missing property name is: name)} {position: line: 3, column: 86, offset: 107} for query: '\n        MATCH (n1)-[r]->(n2)\n        return n1.name AS subject, n1.description AS subject_description, n2.name AS object, n2.description AS object_description, type(r) AS relation, r.description AS relation_description\n        SKIP 0\n        LIMIT 100\n        '
and output of  r2r inspect-knowledge-graph:
Time taken: 0.34 seconds {'results': '\n== Graph Statistics ==\nNumber of nodes: 0\nNumber of edges: 0\nNumber of connected components: 0\n\n== Most Central Nodes =='}
To Reproduce
Steps to reproduce the behavior:

docker compose -f compose.yaml -f compose.neo4j.yaml -f compose.postgres.yaml --project-name r2r up -d
upload file
wait
r2r inspect-knowledge-graph

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]. Linux
Browser [e.g. chrome, safari]. Chrome
Version [e.g. 22] Ubuntu 22.04

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

WARNING:neo4j.notifications:Received notification from DBMS server: {severity: WARNING} {code: Neo.ClientNotification.Statement.UnknownPropertyKeyWarning} {category: UNRECOGNIZED} {title: The provided property key is not in the database} {description: One of the property names in your query is not available in the database, make sure you didn't misspell it or that the label is available when you run this statement in your application (the missing property name is: name)} {position: line: 3, column: 86, offset: 107} for query: '\n        MATCH (n1)-[r]->(n2)\n        return n1.name AS subject, n1.description AS subject_description, n2.name AS object, n2.description AS object_description, type(r) AS relation, r.description AS relation_description\n        SKIP 0\n        LIMIT 100\n
More logs from r2r-r2r-1

Hey, could you check if your r2r version/docker containers are up to date? Also, it is recommended that you start your server using the following command.
r2r serve --docker --config-name=neo4j_kg

You should be able to do
r2r ingest-sample-file
r2r enrich-graph
r2r inspect-knowledge-graph

And finally, then try with your custom file
Let me know if this doesn't work. Thanks!

The problem is I cannot start like that. I am doing it in air gapped servers. I am using a repository that caches other docker images. Then I need to change docker images.
I tried as you said:
r2r ingest-sample-file
r2r inspect-knowledge-graph

I have the same error.
I started from neo4j_kg config as you mentioned.

@shreyaspimpalgaonkar


Create development milestones and project board

After diagramming and gap analysis (#2), create milestones for major development/delivery phases and kanban-ish project board for tracking progress on issues.

To be clear: this is blocked on waiting until we have the repository finalized since our project board should probably live at the org level rather than the repo level.
I will, however, create the milestones now.


Mountinfo: Fix device number parsing

The (major:minor) are treated as hexadecimals, however they are actually unsigned integers

Hi @chenBenjamin97,
Why did you close the PR?

Hi @dtrugman,
I found an issue with the fix.
The device number (major:minor) is represented using decimal numbers at the mountinfo, but represented as hexadecimals in the maps file, so the current code will cause regression.
I wrote a fix for that, once verified I'll push a PR.

The kernel source code seems to agree with you :)
maps - Uses %02x:%02x
mountinfo - Uses %u:%u
block/dev - Uses print_dev_t which is basically %u:%u
Looks good, thank you again!


feat(FN-692): disable function apps in feature deployments

Introduction :pencil2:
During feature deployment function-app deployments fail so the whole action is failing

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (91637c1) 17.31% compared to head (6834272) 22.30%.
Report is 95 commits behind head on main-stbr-tfm-5.

Additional details and impacted files
@@                 Coverage Diff                 @@
##           main-stbr-tfm-5    #2547      +/-   ##
===================================================
+ Coverage            17.31%   22.30%   +4.99%
===================================================
  Files                 1120      797     -323
  Lines                25591    17867    -7724
  Branches              4101     2543    -1558
===================================================
- Hits                  4432     3986     -446
+ Misses               18423    12443    -5980
+ Partials              2736     1438    -1298

Flag
Coverage Δ

api-test
0.48% <ø> (ø)

ui-test
52.37% <ø> (+1.55%)
:arrow_up:

unit-test
4.78% <ø> (-3.18%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feature/cu 1e0zgwg laser doors light

This will correctly set the behavior of the Dissolve and Light animation when doors open/close.
The code has become quite complex due to the different behaviors a door can have.
Level1 will need to be modified and DoorObstacles and EnemySpawnPoints need to be set accordingly to the test scene of this PR. Also, the Factory zone needs to be set properly and the DoorObstacleFactory renamed to DoorObstacle.

Task linked: CU-1e0zgwg LaserDoors light decrease when deactivating /activating


Login error: No auth token

Hi!
I got the error "No auth token" when I try to authenticate the user. I followed the instructions that appear in this web but I can not dismiss the error. I don't know what's the problem exactly.
With 'local' strategy workflow is ok and scripts executes 'passport.use' function. But with 'jwt' strategy I think when script creates 'new JwtStrategy' and tries to extract auth from header can't do it and shows the error inside 'passport.autheticate'.
There are any error?

Can you provide some example code and perhaps a curl command that illustrates how you are passing the token to the to the strategy?

Solved.
The error was application prepends "Bearer" in JWT token and paassport-jwt was expecting "JWT".
Thanks.


Add links to CFP and Unconference issue

Adding links for relevant session submissions.
/assign @mrbobbytables
/assign @detiber

/lgtm

/lgtm
/approve


feat: Deny HTTP on Karpenter SQS policy

Description
I would like to deny access to AWS SQS through HTTP by AWS SQS access policy due to security reasons.
Motivation and Context
SQS security best practices
https://github.com/aws/karpenter-provider-aws/pull/6395
Breaking Changes
No
How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[x] I have tested and validated these changes using one or more of the provided examples/* projects
[x] I have executed pre-commit run -a on my pull request

This PR is included in version 20.15.0 :tada:


Adding an example iris training pipeline

The issue resolved by this Pull Request:
Resolves #<issue_number>
Description of your changes:

Adding an example iris training pipeline, that we could point to for performance testing, or any other purposes.
Testing instructions

Checklist

[ ] The commits are squashed in a cohesive manner and have meaningful messages.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

Change to PR detected. A new PR build was completed.
A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-569

Closing this PR, duplicated in https://github.com/opendatahub-io/data-science-pipelines-operator/pull/573


Add prompts reference with description for each prompt

Based on discussion in #46 this PR adds some documentation explaining what each prompt means and how it is used in the resulting package template.
I've also added a line linking to this document from the README. I'm not sure if that'll be easily missed, but I also didn't want to overcrowd the README.
I'd appreciate a read through and any feedback on important information I may have missed - in particular for the semi-controversial use_git_tags_for_versioning option.

@tlambert03 I've updated the wording for the use_git_tags_for_versioning prompt, but I wholeheartedly agree with you that we should change the default for that prompt to "y" so that I can change the wording to more strongly recommend the default and move the explanation for choosing "n" down to where it's less likely to confuse new users

Thanks for updates here @DragaDoncila

@tlambert03 I've updated the wording for the use_git_tags_for_versioning prompt, but I wholeheartedly agree with you that we should change the default for that prompt to "y" so that I can change the wording to more strongly recommend the default and move the explanation for choosing "n" down to where it's less likely to confuse new users

I agree with default being y and rewording. We could though merge this PR now as is just to get docs in and then update this when/ if we change default. i'm fine either way

@tlambert03 thank you for adding more detail and clarification for the setuptools-scm versioning choice. I've committed that and I think this is now ready to merge, and we'll make the necessary changes after #48 is merged and we change the default to use git tags.


Support providing MessagingUser passwords using a secret

Description

With #3407 work done, the MessagingUser with authentication type 'password' will have passwords displayed when retrieved. A mechanism allowing users to specify passwords by referencing a secret would allow hiding the password.

Addressed in #4391


[blog] Improve blog post for Premium release

preview: https://deploy-preview-32802--material-ui.netlify.app/blog/premium-plan-release/
Improve Premium release blog post

Add Gifs for Premium features (will probably remove gif Aggregation since it's too early stage).

I have tried to move from gif to a video in d5d4c9d so we can compare the result. See the live preview in https://deploy-preview-32944--material-ui.netlify.app/blog/premium-plan-release/#row-grouping and to compare with https://deploy-preview-32802--material-ui.netlify.app/blog/premium-plan-release/#row-grouping. If happy with it, we can cherry-pick the commit on this PR.

@oliviertassinari, the new videos have much better quality and the sizes are great. We can close this PR and merge #32944.

@joserodolfofreitas The file sizes are about the same, if not less. I have documented the tools that I used to pull this off in https://www.notion.so/mui-org/Blog-247ec2bff5fa46e799ef06a693c94917#a94ea053e56f45eaae1220a868a8782a.


Add Localization: Currently supports both English and Chinese.

I refer to the official website for localization examples of extensions: https://github.com/microsoft/vscode-extension-samples/tree/main/l10n-sample.

I only provided localization support for the text in the view of package.json. Currently, I am not very familiar with the code, so I will process the characters in the code when I have time in the future. If you think it's good, welcome to work with me to localize support for this extension.

Since most of the android cmd show english message only.
Just update the UI part only isn't enough to the non-english users.
I still looking around to improve my project.
Because the project without profits. Thanks for your support and bearing with us


Docker Remote API documentation update

I was confused for the longest time on how to actually use and make requests against the remote API, so I think that it might help for those getting started with it to know how to actually test it out via curl. I added in parts on how to access the remote API via curl against the default unix socket, and also on how to configure the docker daemon to expose the API on a TCP port as well if desired.

Please sign your commits following these rules:
https://github.com/docker/docker/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:
$ git clone -b "patch-1" git@github.com:anonmily/docker.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Ammending updates the existing PR. You DO NOT need to open a new one.

LGTM

ping @anonmily would you mind taking care of @moxiegirl's comment ? :stuck_out_tongue_closed_eyes:

ping @anonmily can you address the comments?

Can somebody please carry or close?

@icecrime @thaJeztah carrying this one in #19720


LWKD should include ShoutOuts

We have a ShoutOuts channel in Slack, where contributors give kudos to others who have done things to make the project better.   While we used to read out the kudos at the Community Meeting, that doesn't exist anymore.
Idea: we should publish an abridged collection of the ShoutOuts at the bottom of each LWKD issue.  This wouldn't be a ton of work, and would help call attention to contributors.

+1. Since we wouldn't have shoutouts every week, we can optionally add the section whenever we have some shoutouts from the Slack channel.

+1
It’d spotlight the people who go above and beyond to improve our project. Plus, it could motivate more collaboration and help us keep up the positive momentum!

Added this week as an experiment.


docs: Mention that security groups must be manually referenced on externally attached EKS managed groups

Description
Improving the docs a little bit, I was bitten hard by the lack of guidance on those two variables.
Motivation and Context
Gives the information to anyone that tries to decouple the main eks module to different managed or self-managed groups.
Breaking Changes
None. Docs only.

isn't this covered by the example already? https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/complete/main.tf#L234-L290

@bryantbiggs yes but it's not clear that is mandatory. more specifically, a complete example doesn't tell what's required and what's optional. I'm changing the non-complete example, because one would assume the Managed Group would well, manage itself.
Extra docs never hurt.

This PR is included in version 18.24.0 :tada:


Add http_max_redirects option

There are times when we do not want the client to follow any redirects. This PR adds a http_max_redirects option (exposed from rest-client) to Kubeclient::Client.
Set http_max_redirects: 0 to not follow any redirects which will raise an
exception.

I'm not sure if exposing as http_max_redirects: <integer> is the best idea. Alternative would be have a simpler http_follow_redirects: <boolean> option.

Thanks!
Does this also cover places we use http gem instead of rest-client — watch, and maybe logs(?)
http gem has max_hops option: https://github.com/httprb/http/blob/da68f40182f4230cad0a43b4196e7906c34516df/lib/http/redirector.rb#L41
rest-client is unmaintained, so it seems likely at some point we'll switch (#237)...
Do all gems support "redirect but only k times"?  Would exposing a boolean be more future-proof?
OTOH, are there situations where one wants to increase default max redirects?  rest-client and httpclient default to 10, http gem to only 5, what if one needs more?
=> I'm fine with exposing an integer knob.  (And if so let's set it to 10 for get/watch parity)

Oh, I did not know about http usage. Will look into that.

watch seems to raise on 30x, so it seems moot to pass in redirect options. This seems to be the only place where http gem is  used
get_pod_logs uses the rest-client gem so it should honor the option.

OTOH, are there situations where one wants to increase default max redirects? rest-client and httpclient default to 10, http gem to only 5, what if one needs more?
=> I'm fine with exposing an integer knob. (And if so let's set it to 10 for get/watch parity)

OK, have updated DEFAULT_HTTP_MAX_REDIRECTS to 10

watch seems to raise on 30x, so it seems moot to pass in redirect options.

Untested, but I think that's the response code of the last request, after the gem handled redirects internally?  And when max_hops is exceeded, it'll raise
TooManyRedirectsError or EndlessRedirectError.  Kubeclient will not see 3xx.
That line goes back to #142 which first switched to http gem.  It was not discussed there.
I dont think < 300  test had any intent to raise on redirects, it's just a way to check for 2xx statuses.

Untested, but I think that's the response code of the last request, after the gem handled redirects internally? And when max_hops is exceeded, it'll raise
TooManyRedirectsError or EndlessRedirectError. Kubeclient will not see 3xx.

Interesting, I have added a test to try this out. Without modifications to WatchStream, it fails. So I think http by default does not follow redirects and then we do that check for <300
  1) Error:
TestPodLog#test_watch_pod_log_with_redirect:
Kubeclient::HttpError: Found
    /Users/tkuah/code/kubeclient/lib/kubeclient/watch_stream.rb:20:in `each'
    /Users/tkuah/code/kubeclient/test/test_pod_log.rb:96:in `with_index'
    /Users/tkuah/code/kubeclient/test/test_pod_log.rb:96:in `test_watch_pod_log_with_redirect'

+
+  def test_watch_pod_log_with_redirect
+    expected_lines = open_test_file('pod_log.txt').read.split("\n")
+
+    stub_request(:get, %r{/namespaces/default/pods/[a-z0-9-]+/log\?.*follow})
+      .to_return(status: 302, headers: { location: 'http://localhost:1234/api/namespaces/default/pods/redis-master-pod/log' })
+
+    client = Kubeclient::Client.new('http://localhost:8080/api/', 'v1')
+    stream = client.watch_pod_log('redis-master-pod', 'default')
+    stream.to_enum.with_index do |notice, index|
+      assert_instance_of(String, notice)
+      assert_equal(expected_lines[index], notice)
+    end
+  end

To confirm, I tried out http in irb:
irb(main):003:0> client = HTTP::Client.new
=> #<HTTP::Client:0x00007fd4533503c0 @default_options=#<HTTP::Options:0x00007fd453350118 @response=:auto, @proxy={}, @timeout_class=HTTP::Timeout::Null, @timeout_options={}, @socket_class=TCPSocket, @nodelay=false, @ssl_socket_class=OpenSSL::SSL::SSLSocket, @ssl={}, @keep_alive_timeout=5, @headers=#<HTTP::Headers {}>, @cookies={}, @encoding=nil, @features={}>, @connection=nil, @state=:clean>
irb(main):004:0> client.request(:get, 'https://httpstat.us/302')
=> #<HTTP::Response/1.1 302 Found {"Cache-Control"=>"private", "Location"=>"https://httpstat.us", "Server"=>"Microsoft-IIS/10.0", "X-Aspnetmvc-Version"=>"5.1", "Access-Control-Allow-Origin"=>"*", "X-Aspnet-Version"=>"4.0.30319", "X-Powered-By"=>"ASP.NET", "Set-Cookie"=>"ARRAffinity=77c477e3e649643e5771873e1a13179fb00983bc73c71e196bf25967fd453df9;Path=/;HttpOnly;Domain=httpstat.us", "Date"=>"Wed, 19 Dec 2018 11:03:21 GMT", "Connection"=>"close", "Content-Length"=>"0"}>

irb(main):005:0> client.follow.request(:get, 'https://httpstat.us/302')
=> #<HTTP::Response/1.1 200 OK {"Cache-Control"=>"private", "Content-Length"=>"7959", "Content-Type"=>"text/html; charset=utf-8", "Server"=>"Microsoft-IIS/10.0", "X-Aspnetmvc-Version"=>"5.1", "Access-Control-Allow-Origin"=>"*", "X-Aspnet-Version"=>"4.0.30319", "X-Powered-By"=>"ASP.NET", "Set-Cookie"=>"ARRAffinity=77c477e3e649643e5771873e1a13179fb00983bc73c71e196bf25967fd453df9;Path=/;HttpOnly;Domain=httpstat.us", "Date"=>"Wed, 19 Dec 2018 11:04:19 GMT", "Connection"=>"close"}>

So to be consistent, we should follow redirects for WatchStream too ? If OK, I can perform the change.

Oh, good findings.
Yes, we want watch to be consistent with other methods
(with one exception: we dont want watch to timeout, deliberately)


Redesigned the Navbar (Issue #73)

Change Summary

Changed Navbar to meet new design requirement. (Approved by client)

Implemented a drop-down menu with transitions

Redesigned mobile menu.

Changed the font.
Made it look similar to the drop-down component in large-screen devices.

Change Form
Fill this up (NA if not available). If a certain criteria is not met, can you please give a reason.

[ ] The pull request title has an issue number
[ ] The change works by "Smoke testing" or quick testing
[ ] The change has tests
[ ] The change has documentation

Looks good for the most part, but needs a couple more changes.

Not sure why .prettierignore is committed if it's empty.

Minifying images generally is better for loading. SVG's can be minified using https://jakearchibald.github.io/svgomg/. Try to get the file size as small as possible.

For readability purposes (not necessary) try to group import statements. From what I've seen in codebases and also what I like to do is:

Library imports at the top (import React from 'react')
absolute imports from the codebase (if configured, import Component from '@helper/component')
relative imports with regard to file (import Component from '../Component)

Alrighty, I've made these changes!
Thanks for the suggestions, I'll keep these in mind :)


Changing Account Domain results in RRSessionException

[Enter feedback here]
If I set the account domain to eastus2.mixedreality.azure.com, the RemoteRenderingCoordinator script throws an error and fails to create or connect to a session. If the account domain is westus2.mixedreality.azure.com it functions fine.
The tutorial page specifically says to change the Account domain to match your settings. So this broke the tutorial for me until I figured out what was triggering the error.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: eb3fc294-6247-9b15-9a2c-0dde9eb86e2e
Version Independent ID: 814b28ae-71cc-4bfd-040e-be6eead22035
Content: Viewing a remotely rendered model - Azure Remote Rendering
Content Source: articles/remote-rendering/tutorials/unity/view-remote-models/view-remote-models.md
Service: azure-remote-rendering
GitHub Login: @michael-house
Microsoft Alias: v-mihous

@ARGs-code - Thanks for brining this to our attention. We will investigate it further and update you shortly.

@ARGs-code Thanks for the feedback!
Could you please let us know any error logs completely or any error log screen grabs ? to better understand the issue.
Please mask/hide any sensitive info while uploading error logs or screen grabs from your machine.

@ARGs-code Could you please refer the Supported Regions for Azure remote Rendering.

If I set the account domain to eastus2.mixedreality.azure.com,....

The official Microsoft Documentation page provides list of supported regions as of now. Which do not include eastus2

Please let us know your use case scenario in this matter, so that we can further help!

@ARGs-code
Just checking back! Please let us know if you need further help.

Well, that is interesting. Thank you @SatishBoddu-MSFT .
I had thought that eastus2 was supported because the model conversion process works with eastus2 and my Azure Remote Rendering asset on Azure runs on eastus2 as well..... so this is confusing to me.
Thank you for pointing out this documentation.
If eastus2 is not supported, should I open up a different ticket since using eastus2 works with the model conversion process and eastus fails? Or is that working as intended?
I get the following error if I try to perform a model conversion on eastus, and the default ARRConfig.json provided by the documentation has eastus2 listed as one of the supported regions.

@ARGs-code The powershell example scripts incorrectly lists "eastus2" as the name of the eastus region service URL. This error will be fixed in an upcoming release. So in the meanwhile please use "eastus2" when using the conversion script, but it will actually use the eastus region.
Sorry for the confusion and thanks for helping us to improve Azure Remote Rendering

We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.


Addition of Trivial Acceptance Tests

Pull Request (PR) description
Addition of trivial acceptance tests

We need a new modulesync to update https://github.com/voxpupuli/puppet-systemd/blob/master/.github/workflows/ci.yml. when the tests are present and modulesync is run, the CI file is updated to run the acceptance tests.

This is what are feared, you very quickly start running to weird things about these hosts:
Error: /Stage[main]/Systemd::Resolved/File[/etc/resolv.conf]/ensure: change from 'file' to 'link' failed: Device or resource busy @ apply2files - /etc/resolv.conf

That;s probably enough in here for now:

I'll pull out the manage_resolv_conf to it's own MR.
Disable the checks on CentOS as there is indeed no resolved to be had.

#256 first for the extra parameter.


csproj conversion aspnetcore/tutorials/your-first-mac-aspnet.md

https://docs.microsoft.com/en-us/aspnet/core/tutorials/your-first-mac-aspnet

@spboyer  can we trade a couple .Net Core conversion issues for this one?

I will be taking care of the xplat items. If there are some .NET Conversion items you want to look at, please assign them to me.

@spboyer  this is for csproj conversion.


messages: fix top margin in flashed messages

closes https://github.com/inveniosoftware/invenio-theme/issues/167

Flash messages are attached to header with no space in between and also the border of the message is not rounded. Is this what we want @lnielsen ? I assume yes but just for the verification :)

There is also another possibility similar to https://zenodo.org:

Here the flashed message is center and dismissals. @lnielsen which one do you prefer?


Packaging Dependencies

Currently ./install_sct uses two package managers: pip and sct_download_data. The latter we are maintaining ourselves, and we use it for dependencies that we "own", namely our large pre-trained machine learning models, our example datasets, and the pre-compiled binary programs from ANTs, Sdika's ctrDetect, and some tools from dev/. Maintaining these and the metadata about them is a manual process documented here:

https://github.com/neuropoly/spinalcordtoolbox/wiki/Misc:-How-to-compile-binaries-(ANTs,-isct)
https://github.com/neuropoly/spinalcordtoolbox/wiki/Misc%3A-How-to-package-data-and-binaries

We do not "own" ANTS nor ctrDetect but we are claiming them as our own because they are not well packaged within the wider world: ctrDetect is just a .tar.gz on a website, with no version history available and no mirror, and ANTS is large and difficult to build and on top of that out-of-date and unmaintained in debian, while we depend on having the latest (or at least very recent) version; plus we modify it by only taking a subset of it (antsApplyTransforms, antsRegistration, antsSliceRegularizedRegistration, ComposeMultiTransform).
Motivated by #947 / forum365 / #2642, where the issue was that we were shipping very old binaries and easily update them, I want to automate building and packaging these dependencies as much as possible.
Here is my plan:

[ ] Sort out, conceptually, the dependency relationships: https://github.com/neuropoly/spinalcordtoolbox/issues/2669#issuecomment-617371216
[ ] Manually normalize current packages

These will become the goal for the automated packages; that way we can distinguish what issues arise due to updating the code or due to the automation

[ ] Update ANTS (conceptually, spinalcordtoolbox-ants): #2642
[ ] Update ctrDetect: #2666

snag: actually a downgrade :/

[ ] Update dev tools (conceptually, spinalcordtoolbox-dev):

snag: I have not been able to build dev/isct_propseg or dev/isct_dice_coefficient; I think they need maintenance, the dates in our current distribution show they have not been rebuilt since the first time they were distributed in 2016.

[ ] Incorporate missing copyright notices: #2667

[ ] Automate packaging in CI

this will be done in isolated repositories, to avoid crashing this repository with broken CI mistakes and superfluous releases

Binaries

[ ] spinalcordtoolbox-ants: https://github.com/kousu/ANTs

snag: not quite working; it's failing on OS X 10.12 and 10.13
side quest: build with -DBUILD_SHARED_LIBS=on to shrink the size; but this makes the distribution require a shared lib, which would be fine if this was a system-wide install but harder

look into how brew handles this

side quest: build only the four programs we need

[ ] spinalcordtoolbox-dev:
[ ] ctrDetect:

snag: currently closed source, but this could change if we just ask nicely. The license is 2-clause BSD, afterall.

[x] spinalcordtoolbox-binaries: https://github.com/kousu/spinalcordtoolbox-binaries

Datasets

[ ] sct_testing_data
[ ] sct_example_data

Models

[ ] PAM50 -> sct-model-PAM50
[ ] MNI-Poly-AMU -> sct-model-MNI-Poly-AMU
[ ] gm_model -> sct-model-gm
[ ] optic_models -> sct-model-optic

Is this actually several models? Maybe they should be separate packages + a metapackage pointing to them?

[ ] ...

[ ] Automate publishing in CI

Publish to:

[x] Github Releases -- automating this is easy (see example)
[ ] https://osf.io/6zbyf/ -- see osfclient
[ ] https://www.neuro.polymtl.ca/_media -- I wrote a script for this

[x] spinalcordtoolbox-ants:  https://github.com/kousu/ANTs/releases
[ ] spinalcordtoolbox-dev
[ ] ctrDetect
[x] spinalcordtoolbox-binaries:  https://github.com/kousu/spinalcordtoolbox-binaries/releases

[ ] Incorporate the CI building into this mainline repo, with each sub-package a different build job

This will make this repo much more noisy, and eat up lots of space; we have to do it carefully.
This will make SCT a multi-package repo; see https://lerna.js.org for inspiration about this; I need to learn more about this before I can do it.

[ ] Split packages up, have the installer install them separately and/or use metapackages.

[ ] spinalcordtoolbox-binaries -> spinalcordtoolbox-ants, spinalcordtoolbox-dev, ctrDetect

Easy, but requires some careful backwards-compatibility work and/or updates to internal code because spinalcordtoolbox-binaries is responsible for adding the "isct_" prefix to all apps, so we'd lose that.

[ ] Split spinalcordtoolbox-dev -> spinalcordtoolbox-dev-propseg, spinalcordtoolbox-dev-dice_coefficient

[ ] Make python wheels

Our trained models are like plugins
Our binaries are like binary extensions, though we'd locate them with pkg_resources instead of import.
[ ] Replace sct_download_data -d $DEPENDENCY -> pip install $DEPENDENCY

[ ] Figure out package signing, or at least checksumming

pip supports this but not very well; and I am unsure how it interacts with per-platform wheels since each platform file will have a different checksum.

[ ] Publish to pypi, so that pip install spinalcordtoolbox works.
[ ] By january 2021: Stop distributing old centos6 package.

Details
We build for two platforms: osx and linux. We are temporarily keeping an old centos6 version too but it is unmaintained, and centos6 is end-of-life in the end of this year. All of these are 64-bit Intel architectures.
linux specifically means centos7_x86_64, aka manylinux2014_x86_64, because that is the most cross-compatible linux out there.
We do not support ARM or 32-bit intel. Maybe some day this packaging infrastructure could be expanded to cover more platforms; the Pypa people have made it work.
Questions:
Q: Why not just commit the dependencies to this repo?
A: They are very very large. It would get unwieldy very quickly to have gigabytes of old versions stored in the .git history. A lot of this work is just to avoid overwhelming size.
Q: Does "automate publishing" mean publishing to Github Releases, OSF.io, www.neuro.polymtl.ca/_media? Does it mean giving credentials to our mirrors to a robot? Is that safe?
A: We can make isolated credentials and store them in Github Secrets, but yes we should be careful.
Q: Do we really need to package spinalcordtoolbox-dev? If it's only for development couldn't we trust that developers can build these apps on their machines?
A: We need to make more accessible and reliable build instructions for these before this is realistic. I don't know yet
Q: What about using distro packages?
A: I like this strategy too. We could take over or start maintainership of ANTS and ctrDetect in Debian, CentOS, and brew, and work with the people in the Arch community who've already been taking care of ANTS there. The trade-off here is the classic tradeoff between distro package managers (apt, brew, pacman, yum, chocolatey, nixos) and language package managers (pip, npm, cpan, gem, maven, arguably docker): with the former there's a wider audience, better integration and testing, but it's slower for us to get updates distributed than with the latter.

There's a useful pro-distro perspective from Drew DeVault
Our trained models don't fit in here.

Q: Could we break ANTS up into subpackages instead of building the whole thing and keeping only what we want?
A: Perhaps, but we would need to work with upstream to do this and this is a pretty invasive change.
Q: Isn't it circular for spinalcordtoolbox to depend on spinalcordtoolbox-dev which is built out of it?
A: Yes. The model we should be going for is a multi-package repo, like https://lerna.js.org/, so that the single repo makes two packages, and that running/installing from source or binary is the same, just the latter is faster. But we're not there yet.
Q: Github Actions vs Travis vs CircleCI
A: I've been using Github Actions to do the build, because it came with easy artifact uploading, and I didn't see the same available in Travis. I saw people suggesting to get an AWS account and use that for artifacts. Also it seems a bit snappier than Travis -- the OS X VMs are much faster to boot -- and it's better integrated into Github, but it is a level of vendor-lock in to worried about and it does change our convention. I do not know what is best.
Q: I have another question!
A: Please post it below!

I think this comment in the wiki is worth preserving, to demonstrate why getting this automated and repeatable (and repeated regularly) is important:

If you want to build your binaries, as of 2017-12-05, you're on your own, but we are working (#1526) on handling bundled vs. system dependencies and building the binaries as part of the installation process (using distribution packages and ultimately setuptools). Feel free to contribute!

Dependency Graph
This is the conceptual dependency graph; eventually I want to encode this as data, using a package management formats (probably requirements.txt but I'm open to suggestions).

TODO

@jcohenadad says that isct_dice_coefficient is unused. So really spinalcordtoolbox-dev is just spinalcordtoolbox-propseg. It still needs to be recompiled.

For reference, the current unfinished procedure for updating ANTs, which is the biggest of these dependencies, is to first merge upstream into https://github.com/kousu/ANTs/tree/matrix-build, wait for the build to run on https://github.com/kousu/ANTs/pull/5 and publish a release to https://github.com/kousu/ANTs/releases, then manually upload the results from /releases to osf.io, and change ./scripts/sct_download_data.py to match to new versions.
I still need to document this better, and migrate that fork to neuropoly/ANTs. It probably also needs to pay closer attention to the git tags, instead of just uploading to a release named after the nonsense sha1sum. When this is actually tidied up, the procedure will be:
# one time setup
git clone git@github.com:neuropoly/ANTs;
cd ANTs
git remote add upstream git@github.com:ANTsX/ANTs.git

# syncing
git checkout build
git rebase upstream/master
git push -f

then wait, then manually upload the result to osf.io.
But I also want to automate it further than that and make sct_download_data.py use Github directly, to cut out  the osf step and maybe make a cronjob to do the syncing.

additional note: for now, the "Linux" binaries to select are the centos7 build.

Adding a note here for https://github.com/pypa/packaging-problems/issues/64, which I think might be worth reading through. It has lots of input from maintainers who are in similar situations as us, as well as from PyPA devs stating their goals/intentions (highlight: https://github.com/pypa/packaging-problems/issues/64#issuecomment-469409655).
(Also, I believe wheels not supporting post-install scripts was the main roadblock @Drulex encountered when experimenting on the aj/pip-only-install branch.)

Responding to https://github.com/neuropoly/spinalcordtoolbox/issues/1526#issuecomment-813085296...

The biggest thing I'm stuck on, and I'm happy to keep chewing on, is how to handle binaries_*. We might have to migrate everything else to pip but keep install_sct around for a while until someone can deal with them

An idea for an incremental approach could be to slowly reduce our dependency on external pre-compiled binaries, and instead more closely integrate the source code into spinalcordtoolbox:

ANTs binaries -> ANTsPy (#2007)
dev/isct_* C++ code -> Cython wrapper (See scikit-learn's guide and the Using C++ in Cython doc page.) This would also be closer to the currently-recommend ways of handling "accelerator modules".
I'm not sure how to handle isct_spine_detect and isct_train_svm, though, because we don't have access to the source code. If we did, we might be able to write Cython wrappers for these, too.

Just avoiding the issue would solve a lot!
spine_detect is just a call to SVM. We should be able to replace that with sklearn.svm. I have been hesitant because numerical stability but I think it's worth it. We can patch up whatever goes awry.
I think ANTsPy is a much bigger project. Worth it, but it's just...huge. ANTs underlies most of the operations in SCT doesn't it? Fiddling with that is going to cause a cascade of changes and numerical issues. I originally figured I would get around to solving this as an incremental approach towards that that but that was overpromising to myself.

Another bit of progress: https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models (I don't have access to https://github.com/sct-data/c2c3_disc_models so I moved this back over here; and anyway it's weird to have it split out over there isn't it?)
I learned a lot about how pip and packages work. A package is just a zipfile that gets unpacked into site-packages/; in particular there's no need for it to constrain itself to only unpacking into one top-level folder. In fact every .whl unpacks at least two folders: site-packages/$pkg/ and site-packages/$pkg-$verison.dist-info/. Furthermore, there's nothing stopping a package from unpacking itself inside of a different package's folder.
And so that's what I've done here. I've moved $SCT_DIR/data/c2c3_disc_models to site-packages/spinalcordtoolbox/data/c2c3_disc_models.
We can access them (and I'll be editing SCT to do this ) with:
with importlib.resources.path('spinalcordtoolbox.data.c2c3_disc_models','t1_model.yml') as path: pass

or even better:
try:
    import spinalcordtoolbox.data.c2c3_disc_models
except ImportError:
    warnings.warn("Missing data needed for ..... run pip install spinalcordtoolbox-data-c2c3_disc_models to get it.")
    raise

...

with importlib.resources.path(spinalcordtoolbox.data.c2c3_disc_models,'t1_model.yml') as path: pass

(the weird with here is going to have to be idiomatic I think: a long time ago packages were defined as EITHER folders OR zipfiles with room left open for further expansion to e.g. loading code live over the web, so you need the with to handle the case where you potentially have temporarily unpacked a virtual file into a real file, but no one uses any other form anymore [1] and I think we should ignore it, hence this idiom; maybe I'll patch importlib.resources)
Unpacking into other package's folders has a nasty quirk though: if you use pip install -e at any point, then you break the easy case and fall into semi-virtual namespace packages, where the package as seen by python is cobbled together from multiple folders. So despite PEP420 there claiming

A namespace package is not fundamentally different from a regular package. It is just a different way of creating packages. Once a namespace package is created, there is no functional difference between it and a regular package.

a. you can't use importlib.resources on namespace packages (it contains a comment buried that "By definition, namespace packages cannot have resources." which is ridiculous because you can scatter python files in any combination across the component pieces of a namespace package and they're all importable, but it blinds itself to non-python files in this case)
b. a "regular package" is one containing a __init__.py, and in that case the scanning algorithm stops immediately, so another difference is that a "regular package" cannot have any child packages unless they are physically in the same folder as it -- so pip install -e spinalcordtoolbox will blind it to everything in spinalcordtoolbox.data; pip install -e spinalcordtoolbox-data-c2c3_disc_models will be okay but only because I'm not planning on putting any child data folders inside of it
To avoid this, I think we should remove all __init__.pys; that's what PEP420 recommends. Except we need to keep them in these data packages, but I can just leave a note there about this weird python bug (and maybe I'll go put it on the python bug tracker and see if I can talk them into changing the API; probably not though, this has already taken a lot of energy).
We could also skip the issue by installing to a separate hierarchy, say, spinalcordtoolbox-data.c2c3_disc_models; then pip install -e spinalcordtoolbox won't accept. Or even just put every data package as its own top-level package: spinalcordtoolbox-data-c2c3_data_models; this is what flask plugins do.
Or we could just not worry about it since it's a rare issue anyway? We could just test with pip install; that might not be so bad? (or..maybe it is; the skimage people were pretty concerned about how inefficient pip is about rebuilds, especially in the 'modern' age of pyproject.toml: https://github.com/scikit-image/scikit-image/pull/4920#issuecomment-676748922 )
I still think this is a good solution and perfectly serviceable for the size of data we're pushing.
[1]

"However, while this behaviour is a natural consequence of the format design, actually relying on it is generally discouraged."
cite

And I got it auto-releasing: https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/blob/master/.github/workflows/release.yml
And even better, check this out, we can keep using Github as a mirror with pip:
Binary from pypi
(.venv) [kousu@requiem c2c3_disc_models]$ pip install --extra-index-url https://test.pypi.org/simple spinalcordtoolbox-data-c2c3-disc-models
Looking in indexes: https://pypi.org/simple, https://test.pypi.org/simple
Collecting spinalcordtoolbox-data-c2c3-disc-models
  Downloading https://test-files.pythonhosted.org/packages/2d/9f/fb2b637a179b71706162595b74293350c3418db877c56ce5430ddca8791d/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl (10 kB)
Installing collected packages: spinalcordtoolbox-data-c2c3-disc-models
Successfully installed spinalcordtoolbox-data-c2c3-disc-models-2019.1.17
(.venv) [kousu@requiem c2c3_disc_models]$ cat .venv/lib/python3.9/site-packages/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD
spinalcordtoolbox/data/c2c3_disc_models/__init__.py,sha256=BiGYrD6AArCZ3GSZTIDBvfEzRkwCX0NQZ6dURjhe4yA,65
spinalcordtoolbox/data/c2c3_disc_models/__pycache__/__init__.cpython-39.pyc,,
spinalcordtoolbox/data/c2c3_disc_models/t1_model.yml,sha256=Z61YZN-zs23kRGPCHe4NZIfN7zOYSE1VybV2ulpq8VA,10933
spinalcordtoolbox/data/c2c3_disc_models/t2_model.yml,sha256=PWFSHRCk52NL29--ScDNK42IRfjwBzH_RegenwziCII,10899
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/METADATA,sha256=nlWC1c9XGpKPz7y0haozWTsfZtcNy0ccBpldjebTF7A,443
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD,,
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/top_level.txt,sha256=Atnb0WhQSzgKr9R4AEJZmOYo8scDV5TtfT1ld0s3MFU,18
(.venv) [kousu@requiem c2c3_disc_models]$ pip uninstall -y spinalcordtoolbox-data-c2c3-disc-models
Found existing installation: spinalcordtoolbox-data-c2c3-disc-models 2019.1.17
Uninstalling spinalcordtoolbox-data-c2c3-disc-models-2019.1.17:
  Successfully uninstalled spinalcordtoolbox-data-c2c3-disc-models-2019.1.17

sdist from pypi
(.venv) [kousu@requiem c2c3_disc_models]$ pip install --no-binary ":all:" --extra-index-url https://test.pypi.org/simple spinalcordtoolbox-data-c2c3-disc-models
Looking in indexes: https://pypi.org/simple, https://test.pypi.org/simple
Collecting spinalcordtoolbox-data-c2c3-disc-models
  Downloading https://test-files.pythonhosted.org/packages/82/8d/c2389f2b9d1e8016450281640fdb84a7f232be8a2684870c2ad4c958be20/spinalcordtoolbox-data-c2c3_disc_models-2019.1.17.tar.gz (11 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: spinalcordtoolbox-data-c2c3-disc-models
  Building wheel for spinalcordtoolbox-data-c2c3-disc-models (PEP 517) ... done
  Created wheel for spinalcordtoolbox-data-c2c3-disc-models: filename=spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl size=10603 sha256=063f6ac2d98c6ad851c5438f4c5a3d21d991a581d0f223d5322bb3333e94dfe4
  Stored in directory: /home/kousu/.cache/pip/wheels/ce/16/de/9005170932196ea2180c3452eee889721483f14879e73c9a50
Successfully built spinalcordtoolbox-data-c2c3-disc-models
Installing collected packages: spinalcordtoolbox-data-c2c3-disc-models
Successfully installed spinalcordtoolbox-data-c2c3-disc-models-2019.1.17
(.venv) [kousu@requiem c2c3_disc_models]$ cat .venv/lib/python3.9/site-packages/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD
spinalcordtoolbox/data/c2c3_disc_models/__init__.py,sha256=BiGYrD6AArCZ3GSZTIDBvfEzRkwCX0NQZ6dURjhe4yA,65
spinalcordtoolbox/data/c2c3_disc_models/__pycache__/__init__.cpython-39.pyc,,
spinalcordtoolbox/data/c2c3_disc_models/t1_model.yml,sha256=Z61YZN-zs23kRGPCHe4NZIfN7zOYSE1VybV2ulpq8VA,10933
spinalcordtoolbox/data/c2c3_disc_models/t2_model.yml,sha256=PWFSHRCk52NL29--ScDNK42IRfjwBzH_RegenwziCII,10899
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/METADATA,sha256=nlWC1c9XGpKPz7y0haozWTsfZtcNy0ccBpldjebTF7A,443
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD,,
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/top_level.txt,sha256=Atnb0WhQSzgKr9R4AEJZmOYo8scDV5TtfT1ld0s3MFU,18
(.venv) [kousu@requiem c2c3_disc_models]$ pip uninstall -y spinalcordtoolbox-data-c2c3-disc-models
Found existing installation: spinalcordtoolbox-data-c2c3-disc-models 2019.1.17
Uninstalling spinalcordtoolbox-data-c2c3-disc-models-2019.1.17:
  Successfully uninstalled spinalcordtoolbox-data-c2c3-disc-models-2019.1.17

Binary from Github:
(.venv) [kousu@requiem c2c3_disc_models]$ pip install -f https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases spinalcordtoolbox-data-c2c3-disc-models
Looking in links: https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases
Collecting spinalcordtoolbox-data-c2c3-disc-models
  Downloading https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases/download/2019.1.17/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl (10 kB)
Installing collected packages: spinalcordtoolbox-data-c2c3-disc-models
Successfully installed spinalcordtoolbox-data-c2c3-disc-models-2019.1.17
(.venv) [kousu@requiem c2c3_disc_models]$ cat .venv/lib/python3.9/site-packages/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD
spinalcordtoolbox/data/c2c3_disc_models/__init__.py,sha256=BiGYrD6AArCZ3GSZTIDBvfEzRkwCX0NQZ6dURjhe4yA,65
spinalcordtoolbox/data/c2c3_disc_models/__pycache__/__init__.cpython-39.pyc,,
spinalcordtoolbox/data/c2c3_disc_models/t1_model.yml,sha256=Z61YZN-zs23kRGPCHe4NZIfN7zOYSE1VybV2ulpq8VA,10933
spinalcordtoolbox/data/c2c3_disc_models/t2_model.yml,sha256=PWFSHRCk52NL29--ScDNK42IRfjwBzH_RegenwziCII,10899
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/METADATA,sha256=nlWC1c9XGpKPz7y0haozWTsfZtcNy0ccBpldjebTF7A,443
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD,,
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/top_level.txt,sha256=Atnb0WhQSzgKr9R4AEJZmOYo8scDV5TtfT1ld0s3MFU,18
(.venv) [kousu@requiem c2c3_disc_models]$ pip uninstall -y spinalcordtoolbox-data-c2c3-disc-models
Found existing installation: spinalcordtoolbox-data-c2c3-disc-models 2019.1.17
Uninstalling spinalcordtoolbox-data-c2c3-disc-models-2019.1.17:
  Successfully uninstalled spinalcordtoolbox-data-c2c3-disc-models-2019.1.17

sdist from Github:
(.venv) [kousu@requiem c2c3_disc_models]$ pip install --no-binary ":all:" -f https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases spinalcordtoolbox-data-c2c3-disc-models
Looking in links: https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases
Collecting spinalcordtoolbox-data-c2c3-disc-models
  Downloading https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases/download/2019.1.17/spinalcordtoolbox-data-c2c3_disc_models-2019.1.17.tar.gz (11 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: spinalcordtoolbox-data-c2c3-disc-models
  Building wheel for spinalcordtoolbox-data-c2c3-disc-models (PEP 517) ... done
  Created wheel for spinalcordtoolbox-data-c2c3-disc-models: filename=spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl size=10603 sha256=db7f4d978e76b499602498eecffd3d48f9f8e82a9e5307327ba5e040e08d3028
  Stored in directory: /home/kousu/.cache/pip/wheels/73/c0/c5/122cb6eef9eccde0bd7345d8ccb6ce1e875e53e1a6060500ca
Successfully built spinalcordtoolbox-data-c2c3-disc-models
Installing collected packages: spinalcordtoolbox-data-c2c3-disc-models
Successfully installed spinalcordtoolbox-data-c2c3-disc-models-2019.1.17
(.venv) [kousu@requiem c2c3_disc_models]$ cat .venv/lib/python3.9/site-packages/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD
spinalcordtoolbox/data/c2c3_disc_models/__init__.py,sha256=BiGYrD6AArCZ3GSZTIDBvfEzRkwCX0NQZ6dURjhe4yA,65
spinalcordtoolbox/data/c2c3_disc_models/__pycache__/__init__.cpython-39.pyc,,
spinalcordtoolbox/data/c2c3_disc_models/t1_model.yml,sha256=Z61YZN-zs23kRGPCHe4NZIfN7zOYSE1VybV2ulpq8VA,10933
spinalcordtoolbox/data/c2c3_disc_models/t2_model.yml,sha256=PWFSHRCk52NL29--ScDNK42IRfjwBzH_RegenwziCII,10899
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/METADATA,sha256=nlWC1c9XGpKPz7y0haozWTsfZtcNy0ccBpldjebTF7A,443
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/RECORD,,
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/REQUESTED,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/WHEEL,sha256=OqRkF0eY5GHssMorFjlbTIq072vpHpF60fIQA6lS9xA,92
spinalcordtoolbox_data_c2c3_disc_models-2019.1.17.dist-info/top_level.txt,sha256=Atnb0WhQSzgKr9R4AEJZmOYo8scDV5TtfT1ld0s3MFU,18
(.venv) [kousu@requiem c2c3_disc_models]$ pip uninstall -y spinalcordtoolbox-data-c2c3-disc-models
Found existing installation: spinalcordtoolbox-data-c2c3-disc-models 2019.1.17
Uninstalling spinalcordtoolbox-data-c2c3-disc-models-2019.1.17:
  Successfully uninstalled spinalcordtoolbox-data-c2c3-disc-models-2019.1.17

With this uploading to pypi, most people will be able to pip install spinalcordtoolbox and have things just work.
For people behind firewalls, we can give them extra mirrors by e.g.:
# sct.txt
-f https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases
-f https://github.com/neuropoly/spinalcordtoolbox/releases
# ... etc
spinalcordtoolbox

And tell people to pip install -r sct.txt. We can get osf.io or other sites in the mix if we just generate a page that looks like https://download.pytorch.org/whl/cpu/torch_stable.html too.

And by the way, that RECORD file is interesting. We angsted a lot over in distriploy over integrity checking. I haven't figured out how to actually make use of it yet, but pip does record hashes of everything it installs. If we can't find a pip verify anywhere we can just write our own.
There's also a way to cryptographically pin the dependencies:
-f https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases
spinalcordtoolbox_data_c2c3_disc_models==2019.1.17 --hash=sha256:29cde6ee062985b05de2f52ee3a1ca2b07b0aaaf4af321cd69fbd3afebfc9cd9

(.venv) [kousu@requiem c2c3_disc_models]$ pip install -r requirements.txt
Looking in links: https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases
Collecting spinalcordtoolbox_data_c2c3_disc_models==2019.1.17
  Downloading https://github.com/neuropoly/spinalcordtoolbox-data-c2c3_disc_models/releases/download/2019.1.17/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl (10 kB)
Installing collected packages: spinalcordtoolbox-data-c2c3-disc-models
Successfully installed spinalcordtoolbox-data-c2c3-disc-models-2019.1.17

but if I instead point it at testpypi, which has an imposter version (under the same tag but uploaded a couple hours ago, and pypi URLs are write-once)
--extra-index-url https://test.pypi.org/simple
spinalcordtoolbox_data_c2c3_disc_models==2019.1.17 --hash=sha256:29cde6ee062985b05de2f52ee3a1ca2b07b0aaaf4af321cd69fbd3afebfc9cd9

(.venv) [kousu@requiem c2c3_disc_models]$ pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://test.pypi.org/simple
Collecting spinalcordtoolbox_data_c2c3_disc_models==2019.1.17
  Downloading https://test-files.pythonhosted.org/packages/2d/9f/fb2b637a179b71706162595b74293350c3418db877c56ce5430ddca8791d/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl (10 kB)
ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    spinalcordtoolbox_data_c2c3_disc_models==2019.1.17 from https://test-files.pythonhosted.org/packages/2d/9f/fb2b637a179b71706162595b74293350c3418db877c56ce5430ddca8791d/spinalcordtoolbox_data_c2c3_disc_models-2019.1.17-py3-none-any.whl#sha256=438d76a4a7504ded1b0e40790d73664ce3422524919fe4c53988d8336b16a2a7 (from -r requirements.txt (line 3)):
        Expected sha256 29cde6ee062985b05de2f52ee3a1ca2b07b0aaaf4af321cd69fbd3afebfc9cd9
             Got        438d76a4a7504ded1b0e40790d73664ce3422524919fe4c53988d8336b16a2a7

I don't think there's a way to do hash-pinning in setup.py, but only in requirements.txt (:shakes fist:). But I would love to be proven wrong about that.

I found more:
https://github.com/neuropoly/spinalcordtoolbox/blob/cf4ed55757107c804b152ac5dc6579c49a54e2a4/spinalcordtoolbox/deepseg/models.py#L20-L80

exvivo_template doesn't seem to be used.

Promising:
kousu@ail:~/src/neuropoly$ pip3 install -i https://test.pypi.org/simple/ spinalcordtoolbox-data-deepseg-sc-models
Collecting spinalcordtoolbox-data-deepseg-sc-models
  Downloading https://test-files.pythonhosted.org/packages/4a/d2/0745b9555aace76fcc51a0cb3e3abf5190490c2e7f683d61936bfc6133db/spinalcordtoolbox_data_deepseg_sc_models-2020.12.3-py3-none-any.whl (58.8MB)
    100% |████████████████████████████████| 58.8MB 8.4kB/s
Installing collected packages: spinalcordtoolbox-data-deepseg-sc-models
Successfully installed spinalcordtoolbox-data-deepseg-sc-models-2020.12.3
kousu@ail:~/src/neuropoly$ python3
Python 3.6.9 (default, Jan 26 2021, 15:33:00)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import spinalcordtoolbox.data.deepseg.sc
>>> import importlib.resources
imTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'importlib.resources'
>>> import importlib_resources
>>> importlib_resources.contents(spinalcordtoolbox.data.deepseg.sc)
['__pycache__', 't2s_ctr.h5', 't1_ctr.h5', 't2s_sc_3D.h5', 'dataset.pkl', 't1_sc.h5', 't2_ctr.h5', '__init__.py', 't2s_sc.h5', 'dwi_sc.h5', 't2_sc_3D.h5', 't2_sc.h5', 'dwi_ctr.h5', 't1_sc_3D.h5']
>>> importlib_resources.contents(spinalcordtoolbox.data.deepseg.sc,'t1_sc.h5')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: contents() takes 1 positional argument but 2 were given
>>> importlib_resources.path(spinalcordtoolbox.data.deepseg.sc,'t1_sc.h5')
<contextlib._GeneratorContextManager object at 0x7f62d2f736a0>
>>> importlib_resources.path(spinalcordtoolbox.data.deepseg.sc,'t1_sc.h5').__enter__()
PosixPath('/home/kousu/.local/lib/python3.6/site-packages/spinalcordtoolbox/data/deepseg/sc/t1_sc.h5')
>>> p = importlib_resources.path(spinalcordtoolbox.data.deepseg.sc,'t1_sc.h5').__enter__()
>>> p.is_file()
True
>>>

(migrated from https://github.com/neuropoly/spinalcordtoolbox/issues/1526#issuecomment-813085296)

Experimentally, Pypi has a 1GB limit on files uploaded to it.
Github's is ~2GB~ 100MB but they only tell you that after you've finished uploading, unless you go over 2GB in which case git just drops you; for anything we can't fit on pypi, for packaging data, we can easily use github with direct links, e.g.
extras_require={
  'PAM50': [spinalcordtoolbox-data-pam50@https://github.com/neuropoly/spinalcordtoolbox/releases/download/5.2.0/pam50.whl],

This should be able to replace most of sct_download_data.

This works for install_requires too, right? I ask because it might be a bit awkward to have to manually specify every data/binary package (e.g. pip install spinalcordtoolbox[PAM50]).

Of course. Of our current pacakges I think
install_requires:

PAM50
gm_model
optic_models
pmj_models
binaries_*
deepseg_gm_models
deepseg_sc_models
deepseg_lesion_models
c2c3_disc_models

extras_require:

sct_example_data
sct_testing_data
MNI-Poly-AMU
course_hawaii17
course_paris18
course_london19
course_bejing19
exvivo_template

And maybe the course_* ones shouldn't be packaged with pip at all, but we should just give people curl commands they can run directly, since they're more of a hands-on sort of thing. Stata has a really neat trick for handling that situation: a Stata package can define "ancillary" files, so that
. net install $pkg

is like pip install $pkg
but
. net get $pkg

gets data files to your working directory. For doing science and tutorials that's really really helpful and I've never seen it built in to another language. But I'm sure we'll figure something out.
The biggest thing I'm stuck on, and I'm happy to keep chewing on, is how to handle binaries_*. We might have to migrate everything else to pip but keep install_sct around for a while until someone can deal with them. See, actually building binaries for multiple platforms is sort of tricky, but I have some of it done in this branch (and have a long plan detailed in #2669). If we get that working, pip can automatically figure out what platform a user is on and finds the right package for you (e.g. https://pypi.org/project/torch/1.5.0/#files covers a subset of {macosx_10_7,macosx_10_6,macosx_10_9,manlinux1}x{x86_64}x{python2.7,python3.5,python3.6,python3.7,python3.8}) so you just say pip install torch and it picks an option for you. Right now we have fragile logic in install_sct to replicate this feature.

(migrated from https://github.com/neuropoly/spinalcordtoolbox/issues/1526#issuecomment-814096301)

gm_model
optic_models
pmj_models
c2c3_disc_models

are really really small. I think we should probably just keep them directly in this repo, under spinalcordtoolbox/data/{gm,optic,pmj,c2c3}.

Food for thought: Should we be packaging the code that uses each of these models with the model?

This line is going to break when I remove init.py
https://github.com/neuropoly/spinalcordtoolbox/blob/3f4b37e09448ba23e9571c4f77645412621c6106/spinalcordtoolbox/compat/launcher.py#L31