'Add new item in visual F# project result in the error below.

Hi @neolem - thanks for the issue report.  Can you provide a repro project and exact steps to make this occur?  Are you using the Visual F# Power Tools? What version of Visual Studio?

Hi @latkin. Yes, I am using the latest version of Visual F# Power Tools(2.0.0) which I updated just about today. So maybe the VFPT has something to do with it as I didn't experience this fault until today. I am also using the just released Visual Studio 2015. I am also using paket generated F# project scaffold.
I got this error after clicking 'Add Above' or 'Add Below' menu options.
I tried to replicate this error in another project but haven't been able to so far. I will update with more information once I can reliably repro this.

closing it for now. Will re-open again if get this issue again.


fix(app): override parts in living as christians part not visible for some users

:tada: This PR is included in version 2.71.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Is the data correct?

Hey Fahd,
I've been trying to work with your data, but I noticed that the numbers don't seem to match the original data available on the PBS website. I consolidated all your files for table 1, and when I calculated the sum (first including "overall" and then removing it), the totals don't align with the national table provided on the PBS website. Specifically, in the "all sex" category, your consolidated dataset sums to 487 million, while the actual value in the PBS PDF is 241 million.
Could you please look into this?

Seems to be coming out fine for me when I sum up the values. There is a small issue in the first table where a tehsil is counted as district due to the way table was processed but if you remove double counts then it is correct. You need to add the column "all_sexes" where region is "OVERALL"

I will adjust the tables to exclude the tehsil level observation being counted as district but overall and mostly, the data is fine

Thanks for starting the issue

No worries, would it be possible for you to proivde me a 1 master file for a specific tablet that has data from all provinces of the table?

I'm having difficulty creating a dashboard from this dataset.
When I filter the entire dataset for table 1 by "overall" and sum up the "all_sexes" column, I get approximately 497 million, which is far from the original figure stated on PBS.

Please let me know if I'm doing anything wrong here.
I'm;

making on masterfile for each table
by adding a province column and conolsidating all the values from rest of the table i.e table01_balochistan_all etc
summing the all_sex values  = 497M = wrong
unpivoting the all_sex value column for better visuals still getting the same value.

Right, so the data ending with the name "_all" contains province, district and tehsil numbers. Summing it up without adding a condition of selecting admin_unit will double or even triple count the data.
If you use Stata by any chance then by using the command:
tabstat all_sexes if region == "OVERALL" & admin_unit == "DISTRICT", stat(sum) format(%13.0f) by(name_admin_unit)
will show you:

Which sums up the total population of Balochistan by district. So just use another filter for admin_unit as you want to count each type of unit uniquely.

The error I found appears in the Sindh data, where Tando Muhammad Khan is being counted twice as a district. This is because Tando Muhammad Khan is both a district and tehsil. So in the district level data, if you remove Tando Muhammad Khan that has the population = 272427, then that will correct this.
There is a chance this is happening in other tables and if you find such double counting of districts then just compare number with original table and remove the one that is erroneously captured.

I will try and correct this bit in the data when i get time and update these datasets. other than that the numbers seem to be alright mainly. But do let me know if you come across other issues as this was a big process.

I figured it out, thanks for your help!


MsalGuard on Redirect URI - getting into redirect loop

Core Library
MSAL.js v1 (@azure/msal or msal)
Core Library Version
1.4.4
Wrapper Library
MSAL Angular (@azure/msal-angular)
Wrapper Library Version
1.1.2
Description
Hello, I'm implementing msal in my angular application and I'm using msal-v1 library.
I have taken angular-7 sample from the documented samples and tried implementing the below code in my enterprise application.
I have added popup as false in app.module.ts
and on app-routing.module.ts, I have added MsalGuard on the empty route as my application wont have any login button and the default landing page also needs authentication.
By doing this, if I directly hit the domain URL(i.e. redirect URI) with out any route,  I was able to authorize and get token properly in the localstorage, but if we clear local storage and then directly navigate to profile path, I was not able to get the token as it is entering a continuously redirect loop.
the application is redirecting to /#id_token=eyJ and then to /profile/#id_token=eyJ and so on.
please guide me on how to guard all routes in my angular application and not enter the redirect loop.
Error Message
No response
Msal Logs
No response
MSAL Configuration
MsalModule.forRoot(
      {
        auth: {
          clientId: "app client id",
          authority:
            "https://login.microsoftonline.com/tenant.onmicrosoft.com/",
          validateAuthority: true,
          redirectUri: window.location.origin,
          postLogoutRedirectUri:
            window.location.origin,
          navigateToLoginRequestUrl: true,
        },
        cache: {
          cacheLocation: "localStorage",
          storeAuthStateInCookie: isIE, // set to true for IE 11
        },
      },
      {
        popUp: false,
        consentScopes: ["user.read", "openid", "profile"],
        unprotectedResources: ["https://www.microsoft.com/en-us/"],
        protectedResourceMap,
        extraQueryParameters: {},
      }
    )

Relevant Code Snippets
const routes: Routes = [
  {
    path: "",
    component: HomeComponent,
    canActivate: [MsalGuard],
  },
  {
    path: "profile",
    component: ProfileComponent,
    canActivate: [MsalGuard],
  },
];

@NgModule({
  imports: [RouterModule.forRoot(routes, { useHash: false })],
  exports: [RouterModule],
})
export class AppRoutingModule {}

Reproduction Steps

refer to Angular-7 Sample in the msal-v1 samples folder.
in app.module.ts - mark popup as false.
add MsalGuard on redirect URI on the routing configuration.
run the application and directly redirect to profile component.

Expected Behavior
get authorized on all routes and get access token without entering the redirect loop.
Identity Provider
Azure AD / MSA
Browsers Affected (Select all that apply)
Chrome
Regression
No response
Source
External (Customer)

@santhosh1248 If you removed the MsalGuard from the default landing page, and directly navigate to the profile path, do you still see this issue?

@jo-arroyo I don't see an issue after removing MsalGuard from the landing page and directly navigate to profile path

@santhosh1248 Having MsalGuard on the landing page is triggering the redirect loops. Is there a reason you chose to use msal-angular v1 and Angular 7? Our recommendation would be to upgrade to msal-angular v2 and follow the instructions here in order to guard the landing page as well.

@jo-arroyo angular version upgrade is scoped for our future releases and for now, our application should be on angular-7 version. is there any way, I could solve this in msal-v1 version?

@santhosh1248 Can you try registering another redirectUri in the Azure Portal (e.g. 'auth-redirect'), setting that as the redirectUri in your app, and creating a new route with that redirectUri?
const routes: Routes = [
  {
    path: '',
    component: HomeComponent,
    canActivate: [
      MsalGuard
    ]
  },
  {
    path: 'auth-redirect',
    component: HomeComponent
  },
  {
    path: 'profile',
    component: ProfileComponent,
    canActivate: [
      MsalGuard
    ]
  }
];

@jo-arroyo it is working fine now, but when authenticating from path other than home page, I see a small UI glitch of home page as the auth-redirect is set to HomeComponent and then to the desired page.
is there anyway we can hide the UI glitch of loading home component?

@santhosh1248 We generally do not recommend setting the MsalGuard on the home page/all paths as looping is an issue. This has been specifically addressed in msal-angular v2 (see the FAQ here), but we are no longer making changes to msal-angular v1. Having another path (auth-redirect) is a workaround, but I admit it may not be perfect. You may wish to investigate if you can have the route without a component, or other strategies.


az interactive ModuleNotFoundError: No module named 'prompt_toolkit.interface'

This is autogenerated. Please review and update as needed.
Describe the bug
Command Name
az interactive
Errors:
The command failed with an unexpected error. Here is the traceback:
No module named 'prompt_toolkit.interface'
Traceback (most recent call last):
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
  File "/usr/local/Cellar/azure-cli/2.41.0/libexec/lib/python3.10/site-packages/azure/cli/command_modules/interactive/custom.py", line 41, in start_shell
    azext_interactive.start_shell(cmd, style=style)
  File "/Users/zhangbo/.azure/cliextensions/interactive/azext_interactive/__init__.py", line 10, in start_shell
    from .azclishell.app import AzInteractiveShell
  File "/Users/zhangbo/.azure/cliextensions/interactive/azext_interactive/azclishell/app.py", line 33, in <module>
    from prompt_toolkit.interface import Application, CommandLineInterface
ModuleNotFoundError: No module named 'prompt_toolkit.interface'

To Reproduce:
Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

Put any pre-requisite steps here...
az interactive

Expected Behavior
Environment Summary
macOS-13.1-x86_64-i386-64bit, Darwin 22.2.0
Python 3.10.8
Installer: HOMEBREW

azure-cli 2.41.0 *

Extensions:
interactive 0.4.6

Dependencies:
msal 1.20.0b1
azure-mgmt-resource 21.1.0b1

Additional Context

Duplicate of https://github.com/Azure/azure-cli/issues/24213


docs: fix repeated typo “heirarchy”

Closes
✅ Pull Request Checklist:

[ ] Included link to corresponding React Spectrum GitHub Issue.
[ ] Added/updated unit tests and storybook for this change (for new code or code which already has tests).
[ ] Filled out test instructions.
[x] Updated documentation (if it already exists for this component).
[ ] Looked at the Accessibility Practices for this feature - Aria Practices

📝 Test Instructions:

🧢 Your Project:

Haha, that's pretty funny how many times we managed to do that, thanks for the PR. Could you follow the instructions here https://react-spectrum.adobe.com/contribute.html#contributor-license-agreement to pass all the PR checks? You may need to close and reopen the PR after following it.

Sure, will look into it…


Gem install fail / Rails 5 / segmentation fault

MacBook High Sierra, V10.13.3, Rails V5.2.0; Ruby V2.4.0p0.
Attempt to bundle installed failed with the following error:
Installing unf_ext <IP_ADDRESS> with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
/Users/jseidel/.rvm/rubies/ruby-2.4.0/bin/ruby -r ./siteconf20180609-26885-1c0cete.rb extconf.rb
checking for -lstdc++... yes
creating Makefile

current directory: /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
make "DESTDIR=" clean

current directory: /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
make "DESTDIR="
compiling unf.cc
make: *** [unf.o] Segmentation fault: 11

make failed, exit code 2

Gem files will remain installed in /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS> for inspection.
Results logged to /Users/jseidel/.rvm/gems/ruby-2.4.0/extensions/x86_64-darwin-15/2.4.0/unf_ext-<IP_ADDRESS>/gem_make.out

An error occurred while installing unf_ext (<IP_ADDRESS>), and Bundler cannot continue.
Make sure that `gem install unf_ext -v '<IP_ADDRESS>'` succeeds before bundling.

In Gemfile:
  attachinary was resolved to 1.3.1, which depends on
    cloudinary was resolved to 1.1.7, which depends on
      rest-client was resolved to 2.0.2, which depends on
        http-cookie was resolved to 1.0.3, which depends on
          domain_name was resolved to 0.5.20180417, which depends on
            unf was resolved to 0.1.4, which depends on
              unf_ext

Log file:
current directory: /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
/Users/jseidel/.rvm/rubies/ruby-2.4.0/bin/ruby -r ./siteconf20180609-26885-1c0cete.rb extconf.rb
checking for -lstdc++... yes
creating Makefile

current directory: /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
make "DESTDIR=" clean

current directory: /Users/jseidel/.rvm/gems/ruby-2.4.0/gems/unf_ext-<IP_ADDRESS>/ext/unf_ext
make "DESTDIR="
compiling unf.cc
make: *** [unf.o] Segmentation fault: 11

make failed, exit code 2

gcc and g++ were causing the problem; symlinked gcc in their place and all is well.


cert-manager http solver pods unable to communicate

We encountered this in the newest CAPA release (0.36.2), but a quick check with newest aws vintage release suggested the same problem occurs there as well so it may be an issue for existing production.
Settings on the WC:

oidc settings on the controlplane were enabled
ingress was added to externalDNS.sources setting for default external dns app
ingress nginx app installed
auth-bundle app installed in kube-system namespace

What happened:
When installing the auth-bundle which contain two ingresses for dex and athena, certificates were not issued by cert-manager as expected.
http solver pods spawned but were unable to communicate through network.
from the challenge resource:
spec:
  authorizationURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/244937906837
  dnsName: dex.aw33u.gaws.gigantic.io
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-giantswarm
  key: gfbAOZpFNZSU-LgxZsp0uoFra-7XeJhMp103BK4Thsw.ve-upqX7AZqNXrbsPPoQghocDAJ8hZSb0fOG6llFjMw
  solver:
    http01:
      ingress:
        class: nginx
  token: gfbAOZpFNZSU-LgxZsp0uoFra-7XeJhMp103BK4Thsw
  type: HTTP-01
  url: https://acme-v02.api.letsencrypt.org/acme/chall-v3/244937906837/gQTlAg
  wildcard: false
status:
  presented: true
  processing: true
  reason: 'Waiting for HTTP-01 challenge propagation: failed to perform self check
    GET request ''http://dex.aw33u.gaws.gigantic.io/.well-known/acme-challenge/gfbAOZpFNZSU-LgxZsp0uoFra-7XeJhMp103BK4Thsw'':
    Get "http://dex.aw33u.gaws.gigantic.io/.well-known/acme-challenge/gfbAOZpFNZSU-LgxZsp0uoFra-7XeJhMp103BK4Thsw":
    context deadline exceeded (Client.Timeout exceeded while awaiting headers)'
  state: pending

Resolver pods were unable to receive requests on the IP addresses, which are assigned to them:
request: "GET /.well-known/acme-challenge/BNslS15jdhLa9YbLuI5KsY7Q1hHzz1FC6DkWLgneDpQ HTTP/1.0", upstream: "http://<IP_ADDRESS>:8089/.well-known/acme-challenge/BNslS15jdhLa9YbLuI5KsY7Q1hHzz1FC6DkWLgneDpQ", host: "athena.rz3s5.gaws.gigantic.io"

When the same endpoint was called from another pod running in the cluster, it did not return any response.
curl http://<IP_ADDRESS>:8089/.well-known/acme-challenge/BNslS15jdhLa9YbLuI5KsY7Q1hHzz1FC6DkWLgneDpQ
curl: (28) Failed to connect to <IP_ADDRESS> port 8089 after 128398 ms: Couldn't connect to server

This seems to be related to default deny-all policy. Applying the following network policy for the specific solver solved the problem.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: http-solver-network-policy-test
  namespace: kube-system
spec:
  egress:
  - {}
  ingress:
  - ports:
    - port: 8089
      protocol: TCP
  podSelector:
    matchLabels:
      acme.cert-manager.io/http-domain: "2701331137"
      acme.cert-manager.io/http-token: "912068148"
      acme.cert-manager.io/http01-solver: "true"
  policyTypes:
  - Ingress
  - Egress

What we expected
When using the default cert manager with the default giantswarm clusterissuer and our ingress nginx app, certificates for ingresses should be issued.

Tried to recreate the issue today on both vintage (gaia) and capi (gollum) using the following steps:

create cluster (vintage: 19.0.0, capi: 0.36.2). Included cert-manager-app chart versions are vintage: 2.21.0 and capi 2.23.2
Install ingress-nginx 2.30.1 via happa
Create a ingress (with changed domain for vintage and capi)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: template-ingress
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-giantswarm
    kubernetes.io/tls-acme: 'true'
spec:
  ingressClassName: nginx
  rules:
    - host: |-
        doing.cluster-id.gaws.gigantic.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kubernetes
                port:
                  name: https
  tls:
    - hosts:
        - doing.cluster-id.gaws.gigantic.io
      secretName: dong-ding

Both times cert-manager was able to issue certificates.
From the linked issue (#2657) I see you installed a second ingress controller.

This was solved by installing the ingress-controller into kube-system and dex and athena into giantswarm namespaces.
Current networkpolicies do not support ingresses in kube-system


All fields are getting added to required

		zodToJsonSchema(z.object({ message: z.string(), data: schema.response.deepPartial() }), {
			$refStrategy: 'none',
		}) as any,

I am using the above code for creating JSON schema for zod schema where all fields are optional. But this code is not working anymore. I tried earlier versions of zod-to-josn-schema, but it's still not working

I found the issue, it seems to be with the zod itself. when we use .and in zod schema it makes fields as required, but when we use .augment it works as expected


Stage import of previous results using transactions

Currently when we import results from previous instances of the course these changes are directly applied onto the DB. This is bad if the file contain any errors, or something else bad happens during upload. We should rather use transactions as provided by MySQL/MariaDB before committing results into production DB.

Extra nice would be if we can preview changes that are about to be applied but this is not mandatory.


Use int_least* family types in minmea_* structs

Closing due to inactivity. If you'd be interested in picking this up we'd be interested in why this change makes sense and a PR if you'd be able.


General tab - redux

Fixes #1262
Reduced version of PR #1267:

Add/rearrange groups on General options tab
Localisations will drop in #1304
Moved What's New button to top of tab

There's no changes to checkboxes or anything like that.
Hopefully small/simple enough PR to sneak in to 11.6.4 release.
build.zip

Localisations, incl. Polish updates, merged. Merging PR to master.


the slammer

all of yall ready to be charged with federal crimes. 15+ years

#please-close

Please explain why I’m being remotely connected to and maliciously attacked seemingly by the same project you guys have going here?

@popebombizzle : Get someone to clean your computer. Your software or browsers seem to be affected by malware.


Add policy pages to footer

Add a link to policy in footer.

@Driesvanransbeeck any action we can already take on this?


pythonPackages.macpack: init at 1.0.3

[X] Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)

[X] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[X] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[X] Fits CONTRIBUTING.md.

@maximedenes Do you want to maintain this package?
@GrahamcOfBorg build python2.pkgs.macpack python3.pkgs.macpack

Success on aarch64-linux (full log)
Attempted: python3.pkgs.macpack
The following builds were skipped because they don't evaluate on aarch64-linux: python2.pkgs.macpack
Partial log (click to expand)
writing top-level names to macpack.egg-info/top_level.txt
reading manifest file 'macpack.egg-info/SOURCES.txt'
writing manifest file 'macpack.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/wv1f0iqh0bhvdr0vajgl8l5rcbdrnhd6-python3.6-macpack-1.0.3

Ran 0 tests in 0.000s

Seems like there are no tests, so please redisable them after all:
# no tests implemented
doCheck = false;

Success on x86_64-linux (full log)
Attempted: python3.pkgs.macpack
The following builds were skipped because they don't evaluate on x86_64-linux: python2.pkgs.macpack
Partial log (click to expand)
writing top-level names to macpack.egg-info/top_level.txt
reading manifest file 'macpack.egg-info/SOURCES.txt'
writing manifest file 'macpack.egg-info/SOURCES.txt'
running build_ext

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
/nix/store/x3b8vjcyhr3algksmh4is8p5jqpzmpm7-python3.6-macpack-1.0.3

This script doesn't seem to fully support a Nix-based setup, please wait before merging, I'll investigate more.

It seems to me that this isn't a library, i.e. it shouldn't be imported using import macpack. Therfore, it should use buildPythonApplication and be moved out of pythonPackages.

It seems to me that this isn't a library, i.e. it shouldn't be imported using import macpack. Therfore, it should use buildPythonApplication and be moved out of pythonPackages.

Yes, that's a good point. Anyway, I realized that the way this utility distinguishes system libraries from user-installed ones seems fundamentally incompatible with a Nix-based layout. So I'm going to close this PR. Sorry for the noise.


Update raycast-arcade extension

Description

Screencast

Checklist

[ ] I read the extension guidelines
[ ] I read the documentation about publishing
[ ] I ran npm run build and tested this distribution build in Raycast
[ ] I checked that files in the assets folder are used by the extension itself
[ ] I checked that assets used by the README are placed outside of the metadata folder

@pernielsentikaer Still same problem :(


Raspberry pi3 avs Build Fail 93%

It long Time to Build And Stop To Build AVS At
[ 93%] Building CXX object Application Utilities/DefaultClient/src/CMakeFiles/DefaultClient.dir/DefaultClient.cpp.o

I have also faced a similar issue this week. As we are using Raspberry Pi 3 Model, it consists of only 1 gb ram and some of the object building processes (like [ 93%] Building CXX object Application Utilities/DefaultClient/src/CMakeFiles/DefaultClient.dir/DefaultClient.cpp.o) from recent avs-device-sdk version requires more memory for object building task.
To solve this issue I increased my swap memory size from 100 mb (default) to 512 mb. After this only my AVS Build process completed. It is not much efficient as it takes a lot of time but it is better than a terminated build.
To increase the swap memory:
Open the terminal, then run

sudo dphys-swapfile swapoff
sudo nano /etc/dphys-swapfile

After the second command, dphys-swapfile file will open and you will need to modify
CONF_SWAPSIZE=100
to
CONF_SWAPSIZE=512
After changing swap size value from 100 to 512
Press CTRL+O then Enter then CTRL+X
(To save the changes made in the file)
Then run
3) sudo dphys-swapfile swapon
Now, reboot or shutdown and start your raspberry pi again and you can see your swap memory size will be increased to 512 mb
(You can check this by running command "free -h" before and after changing the swapsize)
Now, you can run your BUILD AVS process without any error or termination.

Thank You. I Can Run My  Build AVS  Process 100%  And Job Finish But Long Time Not Without Error  And  I Editor Your Script As CONF_SWAPSIZE=1024.It Can Build AVS  To Smooth.

I am glad that setting the swap memory works for you.

I encountered an identical issue. I added a USB thumb drive, noticed that it was setup as /dev/sda, and added /dev/sda to the swap space. After that I had no issues.

I encountered an identical issue. I added a USB thumb drive, noticed that it was setup as /dev/sda, and added /dev/sda to the swap space. After that I had no issues.

It looks like you've resolved this issue, so I'll close this. So far we haven't been able to reproduce the problem on our Pi 3, but we're still investigating. Thanks for letting us know how you fixed it.

It looks like you've resolved this issue, so I'll close this. So far we haven't been able to reproduce the problem on our Pi 3, but we're still investigating. Thanks for letting us know how you fixed it.


Canceled future for execute_request message before replies were done

Environment data

VS Code version: 1.69.2 (user setup)
Jupyter Extension version: v2022.7.1001951036
Python Extension version: v2022.10.1
OS and version: Windows 11 Home 21H2
Python and/or Anaconda version: 3.9.13
Type of virtual environment used: N/A
Jupyter server running: Local

Behaviour
I can reproduce the following error but the context is complex and I haven't been able to isolate it well.
Canceled future for execute_request message before replies were done

The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.

I'm working with a local package. To generate the error, I import the local package in a jupyter cell and call a class method with the wrong name, like this
import pkg
obj = pkg.Obj()
obj.wrong_method_name()

This should generate AttributeError: 'Obj' object has no attribute 'wrong_method_name' but instead it throws the error above. The following generates the correct AttributeError for me:
import pandas as pd
df = pd.DataFrame()
df.wrong_name()

I can generate the same problem using every release of the Jupyter Extension from the last three months (back to 2022.3.x).
Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

info 12:25:35.125: Restart requested C:\DIRECTORY\FILE.IPYNB
info 12:25:35.125: Restart kernel execution
info 12:25:35.125: Cancel pending cells
info 12:25:35.125: Restarting undefined
info 12:25:35.125: Starting raw kernel 'Python 3.9.13' for interpreter c:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.9.exe
info 12:25:35.142: Kernel launching with ports 9005,9006,9007,9008,9009. Start port is 9000
info 12:25:35.156: Process Execution: > ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
> ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
info 12:25:35.162: Creating daemon process for c:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.9.exe with env variables count 49
info 12:25:35.174: Process Execution: > ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_interrupt_daemon -v --ppid 18644
> ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -m vscode_datascience_helpers.daemon --daemon-module=vscode_datascience_helpers.kernel_interrupt_daemon -v --ppid 18644
info 12:25:35.319: get interrupthandle daemon
info 12:25:35.333: Process Execution: > ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -m ipykernel_launcher --ip=<IP_ADDRESS> --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"6dffa17b-5651-46d5-8142-47f401c60b6b" --shell=9007 --transport="tcp" --iopub=9009 --f=c:\Users\user\AppData\Roaming\jupyter\runtime\kernel-v2-18644B7boFiWyTzHr.json
> ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -m ipykernel_launcher --ip=<IP_ADDRESS> --stdin=9008 --control=9006 --hb=9005 --Session.signature_scheme="hmac-sha256" --Session.key=b"6dffa17b-5651-46d5-8142-47f401c60b6b" --shell=9007 --transport="tcp" --iopub=9009 --f=c:\Users\user\AppData\Roaming\jupyter\runtime\kernel-v2-18644B7boFiWyTzHr.json
info 12:25:35.333: Process Execution: cwd: C:\DIRECTORY
cwd: C:\DIRECTORY
info 12:25:35.405: ipykernel version 6.6.0 for c:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.9.exe
info 12:25:35.405: ipykernel location ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\ipykernel\__init__.py for c:\Users\user\AppData\Local\Microsoft\WindowsApps\python3.9.exe
warn 12:25:35.950: StdErr from Kernel Process C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\traitlets\traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\traitlets\traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use '6dffa17b-5651-46d5-8142-47f401c60b6b' instead of 'b"6dffa17b-5651-46d5-8142-47f401c60b6b"'.
  warn(

info 12:25:35.988: Kernel Output: NOTE: When using the `ipython kernel` entry point, Ctrl-C will not work.

To exit, you will have to explicitly quit this process, by either sending
"quit" from a client, or using Ctrl-\ in UNIX-like environments.

To read more about this, see https://github.com/ipython/ipython/issues/2049

To connect another client to this kernel, use:
    --existing c:\Users\user\AppData\Roaming\jupyter\runtime\kernel-v2-18644B7boFiWyTzHr.json

info 12:25:36.193: Got new session b59c6fde-e746-4daf-84a8-a8d61fcc6ed4
info 12:25:36.193: Started new restart session
info 12:25:36.194: Executing silently Code (idle) = import sys\nprint(sys.executable)
info 12:25:36.197: Widget Message: Received IPyWidgetMessages.IPyWidgets_onRestartKernel
info 12:25:36.198: Widget Message: Received IPyWidgets_kernelOptions in ScriptManager
info 12:25:36.198: Widget Message: Received IPyWidgetMessages.IPyWidgets_kernelOptions
info 12:25:36.592: Executing silently Code (completed) = import sys\nprint(sys.executable) with 1 output(s)
info 12:25:36.607: Process Execution: > ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -c "import sys;print(sys.executable)"
> ~\AppData\Local\Microsoft\WindowsApps\python3.9.exe -c "import sys;print(sys.executable)"
info 12:25:36.608: UpdateWorkingDirectoryAndPath in Kernel
info 12:25:36.608: Executing silently Code (idle) = import os\nimport sys\n%cd "C:\DIRECTORY"\nif o
info 12:25:36.624: Executing silently Code (completed) = import os\nimport sys\n%cd "C:\DIRECTORYh"\nif o with 1 output(s)
info 12:25:36.625: Waiting for idle on (kernel): b59c6fde-e746-4daf-84a8-a8d61fcc6ed4 -> idle
info 12:25:36.625: Finished waiting for idle on (kernel): b59c6fde-e746-4daf-84a8-a8d61fcc6ed4 -> idle
info 12:25:39.371: Execute Cell 5 C:\DIRECTORY\FILE.IPYNB
error 12:25:39.714: Disposing session as kernel process died ExitCode:<PHONE_NUMBER>, Reason: C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\traitlets\traitlets.py:2202: FutureWarning: Supporting extra quotes around strings is deprecated in traitlets 5.0. You can use 'hmac-sha256' instead of '"hmac-sha256"' if you require traitlets >=5.
  warn(
C:\Users\user\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\traitlets\traitlets.py:2157: FutureWarning: Supporting extra quotes around Bytes is deprecated in traitlets 5.0. Use '6dffa17b-5651-46d5-8142-47f401c60b6b' instead of 'b"6dffa17b-5651-46d5-8142-47f401c60b6b"'.
  warn(

info 12:25:39.714: kill daemon
error 12:25:39.714: Raw kernel process exited code:<PHONE_NUMBER>
error 12:25:39.715: Error in waiting for cell to complete [Error: Canceled future for execute_request message before replies were done
	at t.KernelShellFutureHandler.dispose (c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:32353)
	at c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:51405
	at Map.forEach (<anonymous>)
	at y._clearKernelState (c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:51390)
	at y.dispose (c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:44872)
	at c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:2366282
	at t.swallowExceptions (c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:7:121082)
	at dispose (c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:2366260)
	at t.RawSession.dispose (c:\Users\user\.vscode\extensions\ms-toolsai.jupyter-2022.7.1001951036\out\extension.node.js:2:2371197)
	at processTicksAndRejections (node:internal/process/task_queues:96:5)]
warn 12:25:39.715: Cell completed with errors {
  message: 'Canceled future for execute_request message before replies were done'
}
info 12:25:39.716: Cancel all remaining cells true || Error || undefined
info 12:25:39.716: Cancel pending cells
info 12:25:39.716: Cell 5 executed with state Error

Thanks for the bug. Something is crashing the kernel.
This is the root cause of the failure:
error 12:25:39.714: Raw kernel process exited code:<PHONE_NUMBER>

Which is a stack overflow error code.
It might be that importing pkg is causing a stack overflow.

If you run that same code in a plain python script, does it throw the attribute error or crash the python process?

The kernel dies silently and sends me back to a terminal when I enter the following lines in iPython
In [1]: import pkg

In [2]: obj = pkg.Obj()

In [3]: obj.wrong_method_name

You can likely debug it using the python debugger. Depends upon if it can run without IPython or not. If you break on all exceptions it might give you an indication where the stack overflow is occurring.

I just remembered I'd written a custom __getattr__ for Obj. That was the source of the problem, no issue with the extension.

We could do a better job translating the error though. Nothing that indicates a stack overflow is anywhere in the cell output.

if it helps, this is a minimal version of my bad code
class Obj():

    def __init__(self):
        self.container = {}

    def __getattr__(self, attr):

        try:
            return getattr(self, attr)

        except AttributeError as error:

            try:
                return self.container['key']

            except KeyError:
                raise error

obj = Obj()
obj.wrong_name


⚠ Bofrost.de has degraded performance

In 3fabd84, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 808 ms

Resolved: Bofrost.de performance has improved in ad37721 after 9 minutes.


[Help] Adding Commands

I wanted to make a other auto commands like ~autohentai, and I believe all my code is correct, but the bot doesn't recognize them as commands.

Then it's not. You must Add the command in the appropriate namespace, and add it to CommandStrings.resx
I don't have time to support people who are editing the source, however you're free to make specific questions about how something works, but it's all pretty simple really.

Thank-you. I forgot to add the commadn to CommandString.resx


remove tests covered by the property testing

Description of changes
Summarize the changes made by this PR.

Improvements & Bug fixes

...

New functionality

...

Test plan
How are these changes tested?

[ ] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes
Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#2789  👈
#2727
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @spikechroma and the rest of your teammates on  Graphite


Inputting nuclei counts into cell2location

Hi, thank you for this tool! I have a question about entering in cell counts. I’m using an older version of cell2location (v.02-alpha) to input in nuclei counts for the 'the expected number of cells per location' hyperparamter. We’re having some trouble getting the latest version (v.0.1.3) to assign cell probabilities to most of the tissue due to high RNA variability after trying both 20 and 200 for alpha (see image below for alpha 200). Areas with low RNA content have very low probabilities assigned for any of the reference cell types. To try to alleviate the problem, we switched to the older version to input custom cell/nuclei counts. In v.02-alpha, I have inputted in a 1-dimensional numpy array with the nuclei counts of each spot (made from concatenating rows of 2d x,y array) on the Visium slide, the following error occurs asking for one value instead of locations specific values: Gamma has no finite default value to use, checked: ('median', 'mean', 'mode'). Pass testval argument or adjust so value is finite.
I tried entering the 2d array directly and got the same error. The model only started to run when I entered one integer, so I was wondering how to input nuclei counts per each spot/location? Any advice on this would be great, thanks!

Here is the model setup:
nuclei_count_1d = np.array(nuclei_counts_1149G['Count'])
r = cell2location.run_cell2location(

      # Single cell reference signatures as pd.DataFrame
      # (could also be data as anndata object for estimating signatures
      #  as cluster average expression - `sc_data=adata_snrna_raw`)
      sc_data=inf_aver,
      # Spatial data as anndata object
      sp_data=slide,

      # the column in sc_data.obs that gives cluster idenitity of each cell
      summ_sc_data_args={'cluster_col': "annotation_1",
                        },

      train_args={'use_raw': True, # By default uses raw slots in both of the input datasets.
                  'n_iter': 40000, # Increase the number of iterations if needed (see QC below)

                  # Whe analysing the data that contains multiple experiments,
                  # cell2location automatically enters the mode which pools information across experiments
                  'sample_name_col': 'sample'}, # Column in sp_data.obs with experiment ID (see above)

      export_args={'path': results_folder, # path where to save results
                   'save_model': True,
                   'run_name_suffix': '' # optinal suffix to modify the name the run
                  },

      model_kwargs={ # Prior on the number of cells, cell types and co-located groups

                    'cell_number_prior': {
                        # - N - the expected number of cells per location:
                        'cells_per_spot': nuclei_count_1d, # < - change this
                        # - A - the expected number of cell types per location (use default):
                        'factors_per_spot': 7,
                        # - Y - the expected number of co-located cell type groups per location (use default):
                        'combs_per_spot': 7
                    },

                     # Prior beliefs on the sensitivity of spatial technology:
                    'gene_level_prior':{
                        # Prior on the mean
                        'mean': 1/2,
                        # Prior on standard deviation,
                        # a good choice of this value should be at least 2 times lower that the mean
                        'sd': 1/4
                    }
      }
)

Hi @ssobt
Looks like the main issue is indeed that a sample has many areas have very low counts. I would address that issue rather than figuring out how to input nuclei counts.

Does nuclei count distribution have a similar distribution to total RNA count? If this is very different it suggests a strong technical effect.
"Areas with low RNA content have very low probabilities assigned for any of the reference cell types." For areas that have little RNA content, as appears to be the case here, you don't necessarily have enough RNA to distinguish any cell type.
Including nuclei count into cell2location would not solve the second problem.
To address, the first problem you could try smaller detection_alpha 0.5 1 2 5. However, if the counts are too low to map anything you would just see noise/similar abundances of all cell types in all low count areas (problem 2).
In projects that I worked on such as (Madissoon et al, we deleted samples with similarly high variability in total count and large areas of low RNA content.
You could try deleting all locations with total RNA count < 1000 or a similar cutoff.

Please don't use older versions (cell2location.run_cell2location) because they are not supported and don't correspond to the published package.

We don't support nuclei count use because we did not find providing that information useful in benchmarks and it is not available for many datasets. Does the analysis work well and provide the expected results with v.02-alpha?
That said, I don't see why providing a 2D shape=(obs, 1) array to N_cells_per_location should be a problem. What error do you see in the latest version?

Actually, I see a problem with the latest version. This line https://github.com/BayraktarLab/cell2location/blob/b2f38944dd13f3b3024e10f97abaf3240f6cccfe/cell2location/models/_cell2location_model.py#L91 needs to be changed to support both scalar values and arrays. Feel free to contribute a PR.

Hi, thanks for the quick response. I don't see why that line in the last comment would cause problems. It would be a scalar divided by an array divided by another scalar so detection_mean_ should come out as an array, which shouldn't be a problem, right?
For some of your questions:

Yes, nuclei count is very similarly distributed to total RNA count
That makes sense that there might not be enough RNA to make distinctions
I figured, but I wanted to give it a try in case it might
I tried running with alphas of 0.5 and 5 (see attached images) and both were very broad in calling 2-3 cell types for all the spots (overlapping calls) hinting to #2 above
Hmm, this is the last resort but certainly on the table if nothing else works
I think given the amount of tissue with lower RNA count (60-70%), it might be easier to simply get more samples

The results for v.02-alpha using a scalar value for 'cells_per_spot' wasn't able to call the low RNA content areas similar to the latest version. I tried running the latest version (v.0.1.3) with a dummy 1d array np.random.randint(1 , 4, 2163) and 2d array np.random.randint(1 , 4, 2163).reshape(2163,1) without any luck. The errors were different each time but I've put them both below. Is there anything I can do to remedy them since I don't think the line you mentioned is what's causing the problem, and thank you for taking such a thorough look at the data.

1d array error output
```
RuntimeError                              Traceback (most recent call last)
File [~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:174](https://vscode-remote+ssh-002dremote-002bnorgay-002eucsf-002eedu.vscode-resource.vscode-cdn.net/home/ssobti/projects/farnaz_spatial/notebooks/monkey_data/cell2location_multi_ref_combined/~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:174), in TraceHandler.__call__(self, *args, **kwargs)
    [173](https://vscode-remote+ssh-002dremote-002bnorgay-002eucsf-002eedu.vscode-resource.vscode-cdn.net/home/ssobti/projects/farnaz_spatial/notebooks/monkey_data/cell2location_multi_ref_combined/~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:173) try:
--> [174](https://vscode-remote+ssh-002dremote-002bnorgay-002eucsf-002eedu.vscode-resource.vscode-cdn.net/home/ssobti/projects/farnaz_spatial/notebooks/monkey_data/cell2location_multi_ref_combined/~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:174)     ret = self.fn(*args, **kwargs)
    [175](https://vscode-remote+ssh-002dremote-002bnorgay-002eucsf-002eedu.vscode-resource.vscode-cdn.net/home/ssobti/projects/farnaz_spatial/notebooks/monkey_data/cell2location_multi_ref_combined/~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:175) except (ValueError, RuntimeError) as e:
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
11 with context:
---> 12     return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
11 with context:
---> 12     return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/nn/module.py:448, in PyroModule.call(self, *args, **kwargs)
447 with self._pyro_context:
--> 448     result = super().call(*args, **kwargs)
449 if (
450     pyro.settings.get("validate_poutine")
451     and not self._pyro_context.active
452     and _is_module_local_param_enabled()
453 ):
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/cell2location/models/_cell2location_module.py:321, in LocationModelLinearDependentWMultiExperimentLocationBackgroundNormLevelGeneAlphaPyroModel.forward(self, x_data, idx, batch_index)
320 shape = self.ones_1_n_groups * b_s_groups_per_location / self.n_groups_tensor
--> 321 rate = self.ones_1_n_groups / (n_s_cells_per_location / b_s_groups_per_location)
322 with obs_plate as ind:
RuntimeError: The size of tensor a (50) must match the size of tensor b (2163) at non-singleton dimension 1
The above exception was the direct cause of the following exception:
RuntimeError                              Traceback (most recent call last)
Cell In[34], line 1
----> 1 mod.train(max_epochs=3000,
2           # train using full data (batch_size=None)
3           batch_size=None,
4           # use all data points in training because
5           # we need to estimate cell abundance at all locations
6           train_size=1,
7           use_gpu=False,
8          )
10 # plot ELBO loss history during training, removing first 100 epochs from the plot
11 mod.plot_history(1000)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/cell2location/models/_cell2location_model.py:209, in Cell2location.train(self, max_epochs, batch_size, train_size, lr, num_particles, scale_elbo, **kwargs)
206         scale_elbo = 1.0 / (self.summary_stats["n_cells"] * self.summary_stats["n_vars"])
207     kwargs["plan_kwargs"]["scale_elbo"] = scale_elbo
--> 209 super().train(**kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/model/base/_pyromixin.py:178, in PyroSviTrainMixin.train(self, max_epochs, use_gpu, train_size, validation_size, batch_size, early_stopping, lr, training_plan, plan_kwargs, **trainer_kwargs)
168 trainer_kwargs["callbacks"].append(PyroJitGuideWarmup())
170 runner = self._train_runner_cls(
171     self,
172     training_plan=training_plan,
(...)
176     **trainer_kwargs,
177 )
--> 178 return runner()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/train/_trainrunner.py:83, in TrainRunner.call(self)
80 if hasattr(self.data_splitter, "n_val"):
81     self.training_plan.n_obs_validation = self.data_splitter.n_val
---> 83 self.trainer.fit(self.training_plan, self.data_splitter)
84 self._update_history()
86 # data splitter only gets these attrs after fit
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/train/_trainer.py:194, in Trainer.fit(self, *args, **kwargs)
188 if isinstance(args[0], PyroTrainingPlan):
189     warnings.filterwarnings(
190         action="ignore",
191         category=UserWarning,
192         message="LightningModule.configure_optimizers returned None",
193     )
--> 194 super().fit(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:608, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
606     raise TypeError(f"Trainer.fit() requires a LightningModule, got: {model.class.qualname}")
607 self.strategy._lightning_module = model
--> 608 call._call_and_handle_interrupt(
609     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
610 )
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py:38, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
36         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
37     else:
---> 38         return trainer_fn(*args, **kwargs)
40 except _TunerExitException:
41     trainer._call_teardown_hook()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:650, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
643 ckpt_path = ckpt_path or self.resume_from_checkpoint
644 self._ckpt_path = self._checkpoint_connector._set_ckpt_path(
645     self.state.fn,
646     ckpt_path,  # type: ignore[arg-type]
647     model_provided=True,
648     model_connected=self.lightning_module is not None,
649 )
--> 650 self._run(model, ckpt_path=self.ckpt_path)
652 assert self.state.stopped
653 self.training = False
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1103, in Trainer._run(self, model, ckpt_path)
1099 self._checkpoint_connector.restore_training_state()
1101 self._checkpoint_connector.resume_end()
-> 1103 results = self._run_stage()
1105 log.detail(f"{self.class.name}: trainer tearing down")
1106 self._teardown()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1182, in Trainer._run_stage(self)
1180 if self.predicting:
1181     return self._run_predict()
-> 1182 self._run_train()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1205, in Trainer._run_train(self)
1202 self.fit_loop.trainer = self
1204 with torch.autograd.set_detect_anomaly(self._detect_anomaly):
-> 1205     self.fit_loop.run()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py:194, in Loop.run(self, *args, **kwargs)
190     return self.on_skip()
192 self.reset()
--> 194 self.on_run_start(*args, **kwargs)
196 while not self.done:
197     try:
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py:217, in FitLoop.on_run_start(self)
214 self._is_fresh_start_epoch = True
215 self._results.to(device=self.trainer.lightning_module.device)
--> 217 self.trainer._call_callback_hooks("on_train_start")
218 self.trainer._call_lightning_module_hook("on_train_start")
219 self.trainer._call_strategy_hook("on_train_start")
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1385, in Trainer._call_callback_hooks(self, hook_name, *args, **kwargs)
1383     if callable(fn):
1384         with self.profiler.profile(f"[Callback]{callback.state_key}.{hook_name}"):
-> 1385             fn(self, self.lightning_module, *args, **kwargs)
1387 if pl_module:
1388     # restore current_fx when nested context
1389     pl_module._current_fx_name = prev_fx_name
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/model/base/_pyromixin.py:45, in PyroJitGuideWarmup.on_train_start(self, trainer, pl_module)
43 tens = {k: t.to(pl_module.device) for k, t in tensors.items()}
44 args, kwargs = pl_module.module._get_fn_args_from_batch(tens)
---> 45 pyro_guide(*args, **kwargs)
46 break
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/nn/module.py:448, in PyroModule.call(self, *args, **kwargs)
446 def call(self, *args, **kwargs):
447     with self._pyro_context:
--> 448         result = super().call(*args, **kwargs)
449     if (
450         pyro.settings.get("validate_poutine")
451         and not self._pyro_context.active
452         and _is_module_local_param_enabled()
453     ):
454         self._check_module_local_param_usage()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:510, in AutoNormal.forward(self, *args, **kwargs)
508 # if we've never run the model before, do so now so we can inspect the model structure
509 if self.prototype_trace is None:
--> 510     self._setup_prototype(*args, **kwargs)
512 plates = self._create_plates(*args, **kwargs)
513 result = {}
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:460, in AutoNormal._setup_prototype(self, *args, **kwargs)
459 def _setup_prototype(self, *args, **kwargs):
--> 460     super()._setup_prototype(*args, **kwargs)
462     self._event_dims = {}
463     self.locs = PyroModule()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:157, in AutoGuide._setup_prototype(self, *args, **kwargs)
154 def _setup_prototype(self, *args, **kwargs):
155     # run the model so we can inspect its structure
156     model = poutine.block(self.model, self._prototype_hide_fn)
--> 157     self.prototype_trace = poutine.block(poutine.trace(model).get_trace)(
158         *args, **kwargs
159     )
160     if self.master is not None:
161         self.master()._check_prototype(self.prototype_trace)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
10 def _context_wrap(context, fn, *args, **kwargs):
11     with context:
---> 12         return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:198, in TraceHandler.get_trace(self, *args, **kwargs)
190 def get_trace(self, *args, **kwargs):
191     """
192     :returns: data structure
193     :rtype: pyro.poutine.Trace
(...)
196     Calls this poutine and returns its trace instead of the function's return value.
197     """
--> 198     self(*args, **kwargs)
199     return self.msngr.get_trace()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:180, in TraceHandler.call(self, *args, **kwargs)
178         exc = exc_type("{}\n{}".format(exc_value, shapes))
179         exc = exc.with_traceback(traceback)
--> 180         raise exc from e
181     self.msngr.trace.add_node(
182         "_RETURN", name="_RETURN", type="return", value=ret
183     )
184 return ret
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:174, in TraceHandler.call(self, *args, **kwargs)
170 self.msngr.trace.add_node(
171     "_INPUT", name="_INPUT", type="args", args=args, kwargs=kwargs
172 )
173 try:
--> 174     ret = self.fn(*args, **kwargs)
175 except (ValueError, RuntimeError) as e:
176     exc_type, exc_value, traceback = sys.exc_info()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
10 def _context_wrap(context, fn, *args, **kwargs):
11     with context:
---> 12         return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
10 def _context_wrap(context, fn, *args, **kwargs):
11     with context:
---> 12         return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/nn/module.py:448, in PyroModule.call(self, *args, **kwargs)
446 def call(self, *args, **kwargs):
447     with self._pyro_context:
--> 448         result = super().call(*args, **kwargs)
449     if (
450         pyro.settings.get("validate_poutine")
451         and not self._pyro_context.active
452         and _is_module_local_param_enabled()
453     ):
454         self._check_module_local_param_usage()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/cell2location/models/_cell2location_module.py:321, in LocationModelLinearDependentWMultiExperimentLocationBackgroundNormLevelGeneAlphaPyroModel.forward(self, x_data, idx, batch_index)
319 # cell group loadings
320 shape = self.ones_1_n_groups * b_s_groups_per_location / self.n_groups_tensor
--> 321 rate = self.ones_1_n_groups / (n_s_cells_per_location / b_s_groups_per_location)
322 with obs_plate as ind:
323     k = "z_sr_groups_factors"
RuntimeError: The size of tensor a (50) must match the size of tensor b (2163) at non-singleton dimension 1
Trace Shapes:
Param Sites:
Sample Sites:
m_g_mean dist           | 1 1
value           | 1 1
m_g_alpha_e_inv dist           | 1 1
value           | 1 1
m_g dist           | 1 9
value           | 1 9
n_s_cells_per_location dist 2163 2163 |
value 2163 2163 |
b_s_groups_per_location dist 2163    1 |
value 2163    1 |
</details>

<details>
<summary>2d array error output</summary>

RuntimeError                              Traceback (most recent call last)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:174, in TraceHandler.call(self, *args, **kwargs)
173 try:
--> 174     ret = self.fn(*args, **kwargs)
175 except (ValueError, RuntimeError) as e:
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
11 with context:
---> 12     return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
11 with context:
---> 12     return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/nn/module.py:448, in PyroModule.call(self, *args, **kwargs)
447 with self._pyro_context:
--> 448     result = super().call(*args, **kwargs)
449 if (
450     pyro.settings.get("validate_poutine")
451     and not self._pyro_context.active
452     and _is_module_local_param_enabled()
453 ):
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/cell2location/models/_cell2location_module.py:390, in LocationModelLinearDependentWMultiExperimentLocationBackgroundNormLevelGeneAlphaPyroModel.forward(self, x_data, idx, batch_index)
382 # =====================Location-specific detection efficiency ======================= #
383 # y_s with hierarchical mean prior
384 detection_mean_y_e = pyro.sample(
385     "detection_mean_y_e",
386     dist.Gamma(
387         self.ones * self.detection_mean_hyp_prior_alpha,
388         self.ones * self.detection_mean_hyp_prior_beta,
389     )
--> 390     .expand([self.n_batch, 1])
391     .to_event(2),
392 )
393 detection_hyp_prior_alpha = pyro.deterministic(
394     "detection_hyp_prior_alpha",
395     self.ones_n_batch_1 * self.detection_hyp_prior_alpha,
396 )
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/distributions/gamma.py:59, in Gamma.expand(self, batch_shape, _instance)
58 batch_shape = torch.Size(batch_shape)
---> 59 new.concentration = self.concentration.expand(batch_shape)
60 new.rate = self.rate.expand(batch_shape)
RuntimeError: The expanded size of the tensor (1) must match the existing size (2163) at non-singleton dimension 0.  Target sizes: [1, 1].  Tensor sizes: [2163, 1]
The above exception was the direct cause of the following exception:
RuntimeError                              Traceback (most recent call last)
Cell In[32], line 1
----> 1 mod.train(max_epochs=3000,
2           # train using full data (batch_size=None)
3           batch_size=None,
4           # use all data points in training because
5           # we need to estimate cell abundance at all locations
6           train_size=1,
7           use_gpu=False,
8          )
10 # plot ELBO loss history during training, removing first 100 epochs from the plot
11 mod.plot_history(1000)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/cell2location/models/_cell2location_model.py:209, in Cell2location.train(self, max_epochs, batch_size, train_size, lr, num_particles, scale_elbo, **kwargs)
206         scale_elbo = 1.0 / (self.summary_stats["n_cells"] * self.summary_stats["n_vars"])
207     kwargs["plan_kwargs"]["scale_elbo"] = scale_elbo
--> 209 super().train(**kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/model/base/_pyromixin.py:178, in PyroSviTrainMixin.train(self, max_epochs, use_gpu, train_size, validation_size, batch_size, early_stopping, lr, training_plan, plan_kwargs, **trainer_kwargs)
168 trainer_kwargs["callbacks"].append(PyroJitGuideWarmup())
170 runner = self._train_runner_cls(
171     self,
172     training_plan=training_plan,
(...)
176     **trainer_kwargs,
177 )
--> 178 return runner()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/train/_trainrunner.py:83, in TrainRunner.call(self)
80 if hasattr(self.data_splitter, "n_val"):
81     self.training_plan.n_obs_validation = self.data_splitter.n_val
---> 83 self.trainer.fit(self.training_plan, self.data_splitter)
84 self._update_history()
86 # data splitter only gets these attrs after fit
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/train/_trainer.py:194, in Trainer.fit(self, *args, **kwargs)
188 if isinstance(args[0], PyroTrainingPlan):
189     warnings.filterwarnings(
190         action="ignore",
191         category=UserWarning,
192         message="LightningModule.configure_optimizers returned None",
193     )
--> 194 super().fit(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:608, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
606     raise TypeError(f"Trainer.fit() requires a LightningModule, got: {model.class.qualname}")
607 self.strategy._lightning_module = model
--> 608 call._call_and_handle_interrupt(
609     self, self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
610 )
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py:38, in _call_and_handle_interrupt(trainer, trainer_fn, *args, **kwargs)
36         return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
37     else:
---> 38         return trainer_fn(*args, **kwargs)
40 except _TunerExitException:
41     trainer._call_teardown_hook()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:650, in Trainer._fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
643 ckpt_path = ckpt_path or self.resume_from_checkpoint
644 self._ckpt_path = self._checkpoint_connector._set_ckpt_path(
645     self.state.fn,
646     ckpt_path,  # type: ignore[arg-type]
647     model_provided=True,
648     model_connected=self.lightning_module is not None,
649 )
--> 650 self._run(model, ckpt_path=self.ckpt_path)
652 assert self.state.stopped
653 self.training = False
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1103, in Trainer._run(self, model, ckpt_path)
1099 self._checkpoint_connector.restore_training_state()
1101 self._checkpoint_connector.resume_end()
-> 1103 results = self._run_stage()
1105 log.detail(f"{self.class.name}: trainer tearing down")
1106 self._teardown()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1182, in Trainer._run_stage(self)
1180 if self.predicting:
1181     return self._run_predict()
-> 1182 self._run_train()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1205, in Trainer._run_train(self)
1202 self.fit_loop.trainer = self
1204 with torch.autograd.set_detect_anomaly(self._detect_anomaly):
-> 1205     self.fit_loop.run()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/loops/loop.py:194, in Loop.run(self, *args, **kwargs)
190     return self.on_skip()
192 self.reset()
--> 194 self.on_run_start(*args, **kwargs)
196 while not self.done:
197     try:
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py:217, in FitLoop.on_run_start(self)
214 self._is_fresh_start_epoch = True
215 self._results.to(device=self.trainer.lightning_module.device)
--> 217 self.trainer._call_callback_hooks("on_train_start")
218 self.trainer._call_lightning_module_hook("on_train_start")
219 self.trainer._call_strategy_hook("on_train_start")
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py:1385, in Trainer._call_callback_hooks(self, hook_name, *args, **kwargs)
1383     if callable(fn):
1384         with self.profiler.profile(f"[Callback]{callback.state_key}.{hook_name}"):
-> 1385             fn(self, self.lightning_module, *args, **kwargs)
1387 if pl_module:
1388     # restore current_fx when nested context
1389     pl_module._current_fx_name = prev_fx_name
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/scvi/model/base/_pyromixin.py:45, in PyroJitGuideWarmup.on_train_start(self, trainer, pl_module)
43 tens = {k: t.to(pl_module.device) for k, t in tensors.items()}
44 args, kwargs = pl_module.module._get_fn_args_from_batch(tens)
---> 45 pyro_guide(*args, **kwargs)
46 break
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/nn/module.py:448, in PyroModule.call(self, *args, **kwargs)
446 def call(self, *args, **kwargs):
447     with self._pyro_context:
--> 448         result = super().call(*args, **kwargs)
449     if (
450         pyro.settings.get("validate_poutine")
451         and not self._pyro_context.active
452         and _is_module_local_param_enabled()
453     ):
454         self._check_module_local_param_usage()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:510, in AutoNormal.forward(self, *args, **kwargs)
508 # if we've never run the model before, do so now so we can inspect the model structure
509 if self.prototype_trace is None:
--> 510     self._setup_prototype(*args, **kwargs)
512 plates = self._create_plates(*args, **kwargs)
513 result = {}
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:460, in AutoNormal._setup_prototype(self, *args, **kwargs)
459 def _setup_prototype(self, *args, **kwargs):
--> 460     super()._setup_prototype(*args, **kwargs)
462     self._event_dims = {}
463     self.locs = PyroModule()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:157, in AutoGuide._setup_prototype(self, *args, **kwargs)
154 def _setup_prototype(self, *args, **kwargs):
155     # run the model so we can inspect its structure
156     model = poutine.block(self.model, self._prototype_hide_fn)
--> 157     self.prototype_trace = poutine.block(poutine.trace(model).get_trace)(
158         *args, **kwargs
159     )
160     if self.master is not None:
161         self.master()._check_prototype(self.prototype_trace)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
10 def _context_wrap(context, fn, *args, **kwargs):
11     with context:
---> 12         return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:198, in TraceHandler.get_trace(self, *args, **kwargs)
190 def get_trace(self, *args, **kwargs):
191     """
192     :returns: data structure
193     :rtype: pyro.poutine.Trace
(...)
196     Calls this poutine and returns its trace instead of the function's return value.
197     """
--> 198     self(*args, **kwargs)
199     return self.msngr.get_trace()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:180, in TraceHandler.call(self, *args, **kwargs)
178         exc = exc_type("{}\n{}".format(exc_value, shapes))
179         exc = exc.with_traceback(traceback)
--> 180         raise exc from e
181     self.msngr.trace.add_node(
182         "_RETURN", name="_RETURN", type="return", value=ret
183     )
184 return ret
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/trace_messenger.py:174, in TraceHandler.call(self, *args, **kwargs)
170 self.msngr.trace.add_node(
171     "_INPUT", name="_INPUT", type="args", args=args, kwargs=kwargs
172 )
173 try:
--> 174     ret = self.fn(*args, **kwargs)
175 except (ValueError, RuntimeError) as e:
176     exc_type, exc_value, traceback = sys.exc_info()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
10 def _context_wrap(context, fn, *args, **kwargs):
11     with context:
---> 12         return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/poutine/messenger.py:12, in _context_wrap(context, fn, *args, **kwargs)
10 def _context_wrap(context, fn, *args, **kwargs):
11     with context:
---> 12         return fn(*args, **kwargs)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/pyro/nn/module.py:448, in PyroModule.call(self, *args, **kwargs)
446 def call(self, *args, **kwargs):
447     with self._pyro_context:
--> 448         result = super().call(*args, **kwargs)
449     if (
450         pyro.settings.get("validate_poutine")
451         and not self._pyro_context.active
452         and _is_module_local_param_enabled()
453     ):
454         self._check_module_local_param_usage()
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/nn/modules/module.py:1194, in Module._call_impl(self, *input, **kwargs)
1190 # If we don't have any hooks, we want to skip the rest of the logic in
1191 # this function, and just call forward.
1192 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1193         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1194     return forward_call(*input, **kwargs)
1195 # Do not call functions when jit is used
1196 full_backward_hooks, non_full_backward_hooks = [], []
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/cell2location/models/_cell2location_module.py:390, in LocationModelLinearDependentWMultiExperimentLocationBackgroundNormLevelGeneAlphaPyroModel.forward(self, x_data, idx, batch_index)
373         pyro.sample(
374             k + "_initial",
375             dist.Gamma(
(...)
379             obs=w_sf,
380         )  # (self.n_obs, self.n_factors)
382 # =====================Location-specific detection efficiency ======================= #
383 # y_s with hierarchical mean prior
384 detection_mean_y_e = pyro.sample(
385     "detection_mean_y_e",
386     dist.Gamma(
387         self.ones * self.detection_mean_hyp_prior_alpha,
388         self.ones * self.detection_mean_hyp_prior_beta,
389     )
--> 390     .expand([self.n_batch, 1])
391     .to_event(2),
392 )
393 detection_hyp_prior_alpha = pyro.deterministic(
394     "detection_hyp_prior_alpha",
395     self.ones_n_batch_1 * self.detection_hyp_prior_alpha,
396 )
398 beta = (obs2sample @ detection_hyp_prior_alpha) / (obs2sample @ detection_mean_y_e)
File ~/miniconda3/envs/cell2loc_env/lib/python3.10/site-packages/torch/distributions/gamma.py:59, in Gamma.expand(self, batch_shape, _instance)
57 new = self._get_checked_instance(Gamma, _instance)
58 batch_shape = torch.Size(batch_shape)
---> 59 new.concentration = self.concentration.expand(batch_shape)
60 new.rate = self.rate.expand(batch_shape)
61 super(Gamma, new).init(batch_shape, validate_args=False)
RuntimeError: The expanded size of the tensor (1) must match the existing size (2163) at non-singleton dimension 0.  Target sizes: [1, 1].  Tensor sizes: [2163, 1]
Trace Shapes:
Param Sites:
Sample Sites:
m_g_mean dist         |  1  1
value         |  1  1
m_g_alpha_e_inv dist         |  1  1
value         |  1  1
m_g dist         |  1  9
value         |  1  9
n_s_cells_per_location dist 2163  1 |
value 2163  1 |
b_s_groups_per_location dist 2163  1 |
value 2163  1 |
z_sr_groups_factors dist 2163 50 |
value 2163 50 |
k_r_factors_per_groups dist         | 50  1
value         | 50  1
x_fr_group2fact dist         | 50 20
value         | 50 20
w_sf dist 2163 20 |
value 2163 20 |
</details>

![image](https://github.com/BayraktarLab/cell2location/assets/101676451/fcd4819d-e0c0-44bf-9f67-e3b9b15f016b)

![image](https://github.com/BayraktarLab/cell2location/assets/101676451/896a6e70-c4ae-4304-a7d8-dd386631ab59)

Whether the problem have be sloved? I am interested in using the N_cells_per_location function by inputing nuclei counts.

Hi @huiyijiangling @ssobt
We are working on incorporating this information at the moment. It is not as simple as changing the above line but requires substantial changes to the model to effectively use segmentation-derived N_cells_per_location. While this will become possible in a month or so - you need to keep in mind that segmentation is not possible for all datasets and it is mostly reliable for FFPE protocols.

Also, when you provide segmentation information detection_alpha has to be large, eg 200.
Segmentation information and large detection_alpha=200 would likely become the new recommended setting.

Hi @vitkl
First of all, thank you for your work !
I'm also interested so If you have any news about your previous comments (the possibility to input the number of cells per-spot instead of a sample-wise value), let us know 😄
Regards,
Benoit

Hi @benoitsam
You can try using this experimental branch https://github.com/BayraktarLab/cell2location/pull/337#issuecomment-2422459954. I am planning to finalise this branch and its dependencies (scvi-tools) by December-February.

Hi @vitkl
Thanks for your reply ! I'll try to follow your instructions to install and use this new feature asap and I'll get back here with the results 😄

Hi @vitkl
Quick comments for the #337 :
Installation
There was an error with the scvi-tools that you forked.
They removed the optional scvi.criticism package so a dirty fix was to do :
git clone https://github.com/vitkl/scvi-tools.git --single-branch --branch pyro_fixes

pyproject.toml : Removed "criticism" line 90

pip install ../scvi-tools
And then for cell2location with your ongoing branch :
pip install "cell2location[tutorials] @ git+https://github.com/BayraktarLab/cell2location.git@hires_sliding_window"
It works if I do import cell2location in python interpreter.
Usage
Could you elaborate on your comment about your N_cells_per_location comment ?
# ideally this is not count of cells
# but % of spot occupied by cells * 0.9999 quantile of N cells across the data

I'm not sure to understand what I'm supposed to use as input because I've got only a count of cells by spot 🤔
Regards,
Benoit

Thanks for suggesting the fix. Good to know.
The idea is that cell abundance is proportional to the number of cells and % of the spot occupied by cells - so combining the two measures gives a better result.
You can use a count of cells by spot too. You need to delete spots with 0 cells.

Hi @benoitsam
It looks like the issue is with posterior sampling rather than training, and you run out of RAM, not GPU memory, right?
The resource change may be due to the new version rather than to using these settings. Do you mean that you are using old parameters with new code?
In general, I would recommend computing quantiles directly like this:
    # In this section, we export the estimated cell abundance (summary of the posterior distribution).
    adata_vis = mod.export_posterior(
        adata_vis, sample_kwargs={
            'batch_size': int(np.ceil(adata_vis.n_obs / 8)),  # this has to be done in batches due to a bug in the code new version
            'accelerator': 'gpu',
            'return_observed': False,
        },
        add_to_obsm=['q05', 'q95', 'q50'],
        use_quantiles=True,
    )

It looks like the issue is with posterior sampling rather than training, and you run out of RAM, not GPU memory, right?

Yes it seems to be an Out of Memory from RAM. I had no warning or error log about GPU memory or CUDA issues.

The resource change may be due to the new version rather than to using these settings. Do you mean that you are using old parameters with new code?

I meant that I used the same "configuration" when I used cell2location (v0.1.4) with N_cells_per_location for the whole sample. "Same configuration" as same cluster, nextflow, SLURM, epoch for training etc...
I tried to use the v0.1.5 by adapting the code and following your comments about the various changes.

In general, I would recommend computing quantiles directly like this:

I tried your suggestion. It works locally with the toy dataset (1700 spots, 10 epoch for the training).
But with the real sample I've got :
Traceback (most recent call last):
    File "/sps/lbmc/bsamson/vap/subworkflows/deconvolution/cell2location/fit_model_prior_by_spot.py", line 210, in <module>
      main()
    File "/sps/lbmc/bsamson/vap/subworkflows/deconvolution/cell2location/fit_model_prior_by_spot.py", line 185, in main
      adata_vis = mod.export_posterior(
    File "/pbs/throng/lbmc/bsamson/software/miniconda3/envs/cell2loc_prior_by_spot_env/lib/python3.10/site-packages/cell2location/models/_cell2location_model.py", line 520, in export_posterior
      self.samples[f"post_sample_{i}"] = self.posterior_quantile(q=q, **sample_kwargs)
    File "/pbs/throng/lbmc/bsamson/software/miniconda3/envs/cell2loc_prior_by_spot_env/lib/python3.10/site-packages/cell2location/models/base/_pyro_mixin.py", line 570, in posterior_quantile
      return self._posterior_quantile_minibatch(exclude_vars=exclude_vars, batch_size=batch_size, **kwargs)
    File "/pbs/throng/lbmc/bsamson/software/miniconda3/envs/cell2loc_prior_by_spot_env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
      return func(*args, **kwargs)
    File "/pbs/throng/lbmc/bsamson/software/miniconda3/envs/cell2loc_prior_by_spot_env/lib/python3.10/site-packages/cell2location/models/base/_pyro_mixin.py", line 444, in _posterior_quantile_minibatch
      valid_sites = self._get_valid_sites(args, kwargs, return_observed=return_observed)
  AttributeError: 'Cell2location' object has no attribute '_get_valid_sites'

At last, it worked on my cluster for real samples 👍

Hi @vitkl !
Do you have any news ? :)


Make updated more resilient

Sleep 30 seconds before kicking off the first updated run cycle. Actually makes the first check faster, since much of the time we were "winning" a race with IsOffroad not yet being set, resulting in a 10 minute wait before the first successful check.
On bootup, after swapping in a finalized overlay upgrade of openpilot, do a git reset --hard of all files. This is a mitigation (not a fix) of the intermittent filesystem damage we've been seeing, wherein random file contents are replaced with the name of a symlink target somewhere else in the openpilot tree. It won't save us from corrupted files in .git, but if that's shown to be a problem, we can explore some repo integrity checks before the overlay swap-in.

I saw a cp -r become a symlink in /data/, if that helps reproduce the intermittent filesystem damage.


min(arg) is an empty sequence -> issues creating analyzer

What is your question?
I am getting the response that min(arg) is an empty sequence when I try and create an analyzer object on a folder. The folder contains one trace in a .json file which was created using the export_chrome_trace function.
The file is quite small, however. ~6.6 kb so I am not sure why the collection is behaving in this manner.
2024-02-23 20:37:50,287 - hta - trace.py:L389 - INFO - traces
2024-02-23 20:37:50,290 - hta - trace_file.py:L61 - ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value.
2024-02-23 20:37:50,291 - hta - trace_file.py:L92 - WARNING - There is no item in the rank to trace file map.
2024-02-23 20:37:50,291 - hta - trace.py:L535 - INFO - ranks=[]
2024-02-23 20:37:50,292 - hta - trace.py:L541 - ERROR - The list of ranks to be parsed is empty.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[27], line 2
      1 from hta.trace_analysis import TraceAnalysis
----> 2 analyzer = TraceAnalysis(trace_dir = "traces")

File /opt/conda/lib/python3.10/site-packages/hta/trace_analysis.py:35, in TraceAnalysis.__init__(self, trace_files, trace_dir)
     29 def __init__(
     30     self,
     31     trace_files: Optional[Dict[int, str]] = None,
     32     trace_dir: str = DEFAULT_TRACE_DIR,
     33 ):
     34     self.t = Trace(trace_files, trace_dir)
---> 35     self.t.load_traces()
     36     assert self.t.is_parsed is True

File /opt/conda/lib/python3.10/site-packages/hta/common/trace.py:414, in Trace.load_traces(self)
    412     return
    413 self.parse_traces()
--> 414 self.align_and_filter_trace()
    415 for rank, trace_df in self.traces.items():
    416     df = self.traces[rank].set_index("index", drop=False)

File /opt/conda/lib/python3.10/site-packages/hta/common/trace.py:552, in Trace.align_and_filter_trace(self)
    548 def align_and_filter_trace(self):
    549     """
    550     Align the starting time across multiple ranks and filter events that belong to incomplete iterations.
    551     """
--> 552     self._align_all_ranks()
    553     self._filter_irrelevant_gpu_kernels()

File /opt/conda/lib/python3.10/site-packages/hta/common/trace.py:646, in Trace._align_all_ranks(self)
    642 def _align_all_ranks(self) -> None:
    643     """
    644     Align dataframes for all ranks such that the earliest event starts at time 0.
    645     """
--> 646     self.min_ts = min(trace_df["ts"].min() for trace_df in self.traces.values())
    647     for rank, trace_df in self.traces.items():
    648         trace_df["ts"] = trace_df["ts"] - self.min_ts

ValueError: min() arg is an empty sequence

I would be grateful for any help that you could provide.
Thank you.
Code
# config from: https://github.com/facebookresearch/HolisticTraceAnalysis/blob/ec3abf17e1afbf2d443e504d0c9a9ee4c9c87a03/examples/cupti_profiler_demo.py#L15
with profile(
  activities = [ProfilerActivity.CUDA, ProfilerActivity.CPU],
  profile_memory = True,
  record_shapes = True,
  with_stack=True,
  experimental_config=torch.profiler._ExperimentalConfig(
            profiler_metrics=[
                # Metrics can be picked from
                # https://docs.nvidia.com/cupti/r_main.html#r_profiler
                # or use kineto__tensor_core_insts, kineto__cuda_core_flops
                "kineto__tensor_core_insts",
                "dram__bytes_read.sum",
                "dram__bytes_write.sum",
            ],
            profiler_measure_per_kernel=True)
) as prof:
    cnn.train() # puts the model into train mode
    print("Starting training")
    for i in range(1, 5):
        train_mnist_for_epoch(i)
        prof.step() # tells the profiler that the steps are correlated
    print("Finished training (need to wait for collection)")
print("Collection finished")

What have you tried?
I have tried both the collection method as described in the read the docs and this example code for a profiler demo.
Environment
OS: Linux (Kaggle)
Python: 3.10.13
HTA version: 0.2.0
Used pip to install

can you please check if the folder has trace files in it?
secondly, please verify that each file has a rank field in it.

if that doesn't help, please share the traces along with the values you used to set for tensorboard_trace_handler and schedule.

I ran the program again and a 1.25 GB source file was produced. The head of the JSON is as follows. rank is contained nowhere in the file.
I was wondering if there was a "hello-world" trace collection program that goes from training a trivial PyTorch model to HTA analyzer?
{
  "schemaVersion": 1,
  "deviceProperties": [
    {
      "id": 0, "name": "Tesla P100-PCIE-16GB", "totalGlobalMem":<PHONE_NUMBER>0,
      "computeMajor": 6, "computeMinor": 0,
      "maxThreadsPerBlock": 1024, "maxThreadsPerMultiprocessor": 2048,
      "regsPerBlock": 65536, "regsPerMultiprocessor": 65536, "warpSize": 32,
      "sharedMemPerBlock": 49152, "sharedMemPerMultiprocessor": 65536,
      "numSms": 56, "sharedMemPerBlockOptin": 49152
    }
  ],
  "record_shapes": 1,
  "with_stack": 1,
  "profile_memory": 1,
  "traceEvents": [
  {
    "ph": "X", "cat": "cpu_op", "name": "autograd::engine::evaluate_function: NllLossBackward0", "pid": 26, "tid": 98,
    "ts":<PHONE_NUMBER>381804, "dur": 7933,
    "args": {
      "External id": 513,"Ev Idx": 0, "Fwd thread id": 1, "Sequence number": 10
    }
  }]
}

The trace can be opened in Peretto. Here is the basic page

@anupambhatnagar Do you have any updates on this?

Another thing you can try is to make sure that the folder which has the trace file has no other files. also the extension of the file should be .json or .json.gz. I wouldn't be able to assist further without the trace file.

There are not other files in the folder and the extension is .json


[Request] Optional Hero Point Rule

Most of my groups ignore nat 1s on hero point rerolls, would it be possible to add a checkbox like the +10 optional rule already in?

We only implement official variants in the system, this would need to be a module.


A few questions before using it

Good afternoon, @hschimpf
First of all, I want to say thank you for your work. It looks very good.
I am very interested in your project as a lightweight alternative to amphp/parallel. It pulls a lot of dependencies, and I really don't like it.
I'm seriously considering it for my project https://github.com/JBZoo/Csv-Blueprint
Exploring your code, I have a few questions:

Why do I need to use a separate loader? I've looked at the references in your readme, but it's still not clear to me.

I noticed that you haven't run [autotests in GA]https://github.com/hschimpf/parallel-sdk/actions for a long time, although the last changes were a few days ago. Of course, on my local, I have everything working fine, and the tests have been completed successfully. Is this how it should be?

Are you using a static code analyzer, such as phptan, psalm, phan, etc? I just didn't see it as part of your GA workflow.

What about PSR12?

Do you have any autotests to check that everything works as expected if the ext-parallel module is not loaded?

Sorry for these questions. It's just that before I use something, I thoroughly research libraries to be sure of my choices for the future.
Thank you in advance.
Best regards, Denis.

In fact, if you don't have time for these improvements, let me know. I'll just make a fork.

Oh, great.
Thanks more for the detailed answer. Honestly, I didn't even think to get that detailed.
I think I will start using the SDK at my place soon. I might have some new issues. I'll be sure to make a new Github issue or something for them.
Yes, I especially like that the library will decide how to run tasks, in a common thread or in parallel (depending on ext-parallel availability). It's simple, easy and doesn't pull a dozen dependencies.
Thanks a lot, @hschimpf! I will look forward to my first experiments.


Selection of Review form after case creation

Hi Everyone,
Not able to create the review form after creation of Case form in service module:
XrmBrowser.Entity.SelectForm("Review", PreThinkTime(7000)) is not working
Steps:

Go to Case
Select new case
Select create form
'XrmBrowser.Entity.SelectForm("Create", PreThinkTime(7000))
Save the case form
Try to select Review form
'XrmBrowser.Entity.SelectForm("Review", PreThinkTime(7000))
Here not selecting the Review form.

How to select Review form after case creation?
Thanks in advance!

Can you please provide the code for your test and we'll try to repro the issue?
Thanks

Thanks for reply.
Below are the code snippets which we have used to select the Review form after creation of case.

Creating the new case
public override bool Execute(XrmBrowser xrmBrowser)
{
string Command = "New case"
xrmBrowser.CommandBar.ClickCommand(Command);
xrmBrowser.ThinkTime(PostThinkTime(1000));
return true;
}

Creating the form
//It is working fine here
public override bool Execute(XrmBrowser xrmBrowser)
{
string Form = "Create";
xrmBrowser.Entity.SelectForm(Form, PreThinkTime(7000));
xrmBrowser.ThinkTime(PostThinkTime(0));
return true;
}

Saving the case
public override bool Execute(XrmBrowser xrmBrowser)
{
xrmBrowser.Entity.Save(PreThinkTime(2000));
xrmBrowser.ThinkTime(PostThinkTime(1000));
return true;
}

Trying to change the form from Create to Review (used same code as above point number 2, only string value Form has changed to Review)
//IT IS NOT WORKING i.e. not opening the dropdown (xpath of dropdown is '{ "Entity_SelectForm",  "id("formselectorcontainer")"}')
public override bool Execute(XrmBrowser xrmBrowser)
{
string Form = "Review";
xrmBrowser.Entity.SelectForm(Form, PreThinkTime(7000));
xrmBrowser.ThinkTime(PostThinkTime(0));
return true;
}

Please let u know if you need any information.
-Thanks

@kishantt Please try using the following code, which is working as expected. My instance does not have a "Create" or "Review" form sections. My form sections are "Summary" and I used "Social Details" respectively. However, it's working as expected and I'm using version 8.2. What version are you using?
The XPath to the flyout form section container: //*[@id="flyoutFormSection_Container"]
using (var xrmBrowser = new XrmBrowser(TestSettings.Options))
{
xrmBrowser.LoginPage.Login(_xrmUri, _username, _password);
xrmBrowser.GuidedHelp.CloseGuidedHelp();
            xrmBrowser.ThinkTime(500);
            xrmBrowser.Navigation.OpenSubArea("Service", "Cases");

            string Command = "New case";
            xrmBrowser.CommandBar.ClickCommand(Command);
            xrmBrowser.ThinkTime(1000);

            string Form = "Summary";
            xrmBrowser.Entity.SelectForm(Form, 1000);

            xrmBrowser.Entity.SetValue("title", "TestReviewForm");
            xrmBrowser.Entity.SelectLookup("customerid", 0);
            xrmBrowser.Entity.Save(2000);

            xrmBrowser.Entity.SelectForm("Social Details", 1000);
        }

@kishantt closing issue. Please open new issue if code provided above doesn't resolve the issue for you. Thank you.