Marketplace is offline

VSCode marketplace is completely down for me.
https://marketplace.visualstudio.com/VSCode - gives me this message

Can you please investigate?

same what happened?

same here, looks like it's down

The server died from the looks of it, It will probably be fixed within a few hours or a day or so.

I also just noticed it. Good luck with fixing it Microsoft!
I'm confident it'll be fixed in a matter of hours 🙂

@RevoluPowered It's clearly a broken robot, his wires are hanging out.

@dapen85 Obviously, the front fell off.

Damn, have neem toying with the idea of getting into QSharp, went to download it and there's a broken robot. The Multiverse is trying to tell me that Im not in a suitable superposition

I reported it to VSTS on twitter too, so it should eventually be fixed lol, I don't think they're checking github atm.

"Apologies for the inconveniences, Gordon.
Our team is already working on this issue. Stay tuned for the latest updates here: http://msft.social/wApOP9 .
Regards."

They should at least implement something in VSCode saying the server is down or something, because right now, you just see that there is no results for your search terms.

They definitely do need to.
**It's pretty serious whats going on: (will definitely have broken other stuff too - on azure) **
Users located in South Central US will be unable to access their account until the underlying issue is resolved. Other regions may experience high failure rates and load times due to dependencies in South Central US. We are working to mitigate the impact for users outside of South Central US to the best of our abilities.

Same here in South Africa.  And this on the day that I need to preview some markdown in the browser!
OS X 10.13.6 and VSCode Version 1.26.1, Johannesburg, ZA for reference.

I badly need Prettier. To bad the marketplace is down :package:

can't believe it down LUL

It's effecting everyone. No updates.
Offic365, Azure, VSCode, VS Team Services, Windows Update will probably be effected too.
Reports in my office of their other platforms slowing down (Azure, Office365) because of the cooling failure in one data center.
Checked with my sysadmin and he is experiencing issues with it too.

Inspecting marketplace API responses for visual studio code looks bad:

Do you think they're potentially causing VSCode to point at another domain
if so, should we even have it open until this is fixed..?

@mglgmz FYI the "malicious" key in the response appears to be a long-standing feature for blacklisting scams, etc: https://github.com/Microsoft/vscode/issues/42115 https://github.com/Microsoft/vscode/issues/41608

ah right, that's good to know.

Just installed Visual Code for the first time and this happened 😄.
Since this issue has been going on since more than 5 hours, I am guessing it's a big one. I will just wait till tomorrow and keep an eye on this thread.

Probably related to Azure problems:
https://azure.microsoft.com/en-us/status/

Starting at 09:29 UTC on 04 Sep 2018, customers in South Central US may experience difficulties connecting to resources hosted in this region. Engineers have isolated an issue with cooling in one part of the data center, which caused a localized spike in temperature, as the preliminary root-cause, which has now been mitigated. Automated data center procedures to ensure data and hardware integrity went into effect when temperatures hit a specified threshold and critical hardware entered a structured power down process.  Engineers are now in the process of restoring power to affected devices as part of the ongoing mitigation process.
Some services may also be experiencing intermittent authentication issues due to downstream Azure Active Directory impact, and engineers are separately working on mitigation options for this also.

@cspotcode yes, I was just freaking out about that one. thanks.

@mayuroks Same =P

+1.

 Saved by the migration :)

+1

Any plans to implement this https://github.com/Microsoft/vscode/issues/57851#issuecomment-418376651

It is funny that Azure AD being down will affect almost all services

Installing Updates ... this may take a while

@issafram It really outlines how much AD has not moved on since windows NT 4.0 was released.
It's a shame the technology hasn't been upgraded to support multiple master nodes, we had this issue at work, our forest main DC died, and we had to do horrible hacks to promote another server to master.
it has no real good redundancy in this regard.
I wonder if they will have troubles getting the AD services back up, because of this issue.

same no ext found !! waht's going on

a localized spike in temperature
...? It was Labor Day.

MS Employee: "Sorry, was in the data-centre and was using one of the racks to cook my sausages, it said "Primary AD NODE", it was really hot so I used that."

I need a plugin for vue xD and.. it's down..  Guess I'll code without format :D

@mayuroks what did you do !? 😱

I think we should make a sister vscode project, with all the plugins while microsoft resolves this 'issue'.
A plugin to download decentralised plugins from another server would be great.
We could host it on AWS and Google cloud 😆 🤣

when is this going to be fix?

Scotty: "our 500 watt electric coolers cant take take it captain, we have too many servers"
Kirk: "dammit scotty, put more servers in the data center; we need more power!"
Scotty: "she cannie take it captain!"
Kirk: "scotty i believe in your abilities"

They should host the Azure Status site on AWS or Google Cloud.  500 errors on status pages are always funny

Does anyone know of where the packages are stored?
We could potentially provide a link for people to access them and install them manually?

"Mitigation efforts continue. Services in South Central US and other regions may experience impact. Engineers are seeing signs of recovery for some services. Please refer to your Portal - http://aka.ms/uafpoewF  Status Page -  http://status.azure.com  and/or Twitter for updates."
(translation)
AD services still fucked, nothing has changed, MS worries about their share price.

If you need to get your hands on an extension you can potentially use this old method.

Find the github repo of the plugin you need.
Go to releases
Download the vsix file from the (stable) release.

This will not work for all plugins, because they might be on other sources
C++ tools can be found here:
https://github.com/Microsoft/vscode-cpptools/releases
Python plugin can be found here:
https://github.com/Microsoft/vscode-python/releases
Omnisharp (C#) plugin can be found here.
https://github.com/OmniSharp/omnisharp-vscode/releases

@mayuroks So code's marketplace went down because you came onboard. 😂

They should really start delivering on that cloud data center in the ocean.
Won't get any overheating issues.

Further on this..
sometimes clicking that file doesn't work.
You have to go into vscode and hit install from VSIX.

@RevoluPowered Thanks for the advice! it worked for the plugin I needed

@MiguelArmendariz glad that it worked :)

I'm opening VSC after like 5 months, just to install a new theme... but guess what, exactly at that moment they are facing problems... It happens the same when I used npm... :(  I should probably stop using technology :-|

You could find the theme's github and do my workaround above @vsambor

And there I was uninstalling and re-installing, resetting my connections and some other witchcraft  lol

@mkisp here is the plugin URL for the prettier thing.
https://github.com/prettier/prettier
Unfortunately they don't have a plugin available to download from their repository.

Thank good for Google and "Install from VSIX", also, for Microsoft publishing their releases to GitHub as well.
Hope you get your poor Azure servers fixed soon! :+1:

So this is not going to be fix today I guess?

lolz man Im just in the setup section  of a course .... and cant install the required plugins xD

@daiky00 https://blogs.msdn.microsoft.com/vsoservice/?p=17405&utm_source=Direct
Next update is in 14 minutes.

@medy111 Which plugins?  You might be able to download and install the .vsix files manually

Seems marketplace is down for me too, I'm trying to install Simple React Snippets and Prettier plugins. Any timeline as to how this might be fixed?

I just built the Prettier plugin.  Download the source code for the VSCode version (I Used 1.6.1).  Open the folder in VSCode.  Open the terminal.  Type npm install, then npm install -g vsce, then vsce package
It will create the vsix package you can install temporarily.

Some plugins you can just download the entire repo files, and unpack it in ~/.vscode/extensions/*/
E.g., https://github.com/ronnidc/vscode-nunjucks

Same here (Brazil).
No extensions are found :cry:

Perhaps I'm ignorant but I'd have thought an issue like this would have been resolved much faster considering the size and resources of Microsoft.

It is funny because they made their distributed services dependent on
non-regional services. AD breaks and all dependent resources as well.
IMO poor design
On Tue, Sep 4, 2018, 7:11 PM Bradley Varol<EMAIL_ADDRESS>wrote:

Perhaps I'm ignorant but I'd have thought an issue like this would have
been resolved much faster considering the size and resources of Microsoft.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vscode/issues/57851#issuecomment-418546429,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAnr4w8Z4wyE_epgO2rsSZj9WDbjWIYwks5uXwiRgaJpZM4WYsRZ
.

How long will it take

LOL. people believe MS can fix this in few mins. However it already take few hours. The problem still there. and one of my Azure app service down as well.

It's hard for MS to maintain a software :)
Sucks!

Over 15 hours... still not working :)

Over 15 hours... still not working :(

I guess I gotta fire up atom...

There are only a couple reasons I can think of why this issue is happening and is not fixed still:

Microsoft tried to test Windows 11
A small drop of water from an employee fell between the cracks of the "Marketplace" mainframe and caused a "class 4 catastrophe"

How long does it takes to recover ?

Update?

+1008611 same here.

can someone share how to share plugin to other vscode? can't wait for fix now..

I can't download about python plugins,I hought is my VPN problem.

still down :( , throw the redundancy out the (ms) window ;)

@zigang93 There is a solution, just on windows, but I think it's same to Mac OS. Copy C:\Users\your name.vscode\extensions all folders to other vscode's extensions folder. Hope to help you.

@justnowanna thanks. will try it out

@Subwater can check the update from here : https://azure.microsoft.com/en-us/status/
or follow vscode at twitter

Still offline in Australia too.
I downloaded the extension I needed from github and added to .vscode/extensions.
Looks fine

Okay... so it's still down, should I use Sublime Text in the mean time?

marketplace is up now.Extension is working...

Well, I thought it was up...  oh... OK, now it is.

Hi 👋
Our marketplace is slowly coming back. You should be able to search and download extensions again.
We are terrible sorry for the inconvenience this outage has caused.
We'll be following up with a post-mortem of this incident soon.
/k

Marketplace still cannot be accessed (as of 5 Sept 2018/Wed)!

The marketplace is up now. I can able to install extensions :)

Yes, facing the same problem.

I can access the marketplace now at https://marketplace.visualstudio.com/VSCode. But cant search for any extensions in VS code itself. Running on macOS High Sierra

i need the extensions :( But thank you for confirming

Hmm, seems like search API is broken at the moment. Cant search on the website or app

@rumbles404 if you need extensions, you can install it via the web for now. Just try google 'extension-name vs code marketplace' and install from there

so its not installation problem at my end .. Server end problem.

So @auchenberg any chance you can implement https://github.com/Microsoft/vscode/issues/57851#issuecomment-418376651 into VSCode ?

https://github.com/Microsoft/vscode/issues/57901

Is this problem resolved?

The problem still exists 'No extensions found' - fedora 27.

Actually I'm facing the same problem on macOS 10.13.6 , Code Version 1.27.2, no errors in any logfiles.


Update starlette to 0.27.0

Fix pip-audit for admin and api
Found 1 known vulnerability in 1 package
Name      Version ID                  Fix Versions
--------- ------- ------------------- ------------
starlette 0.25.0  GHSA-v5gw-mw7f-84px 0.27.0

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.


Button styling update

-Remove form io button radius:0 as it prevents us from styling their components
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the met-public license (Apache 2.0).

"-Remove form io button radius:0 as it prevents us from styling their components"
their? which components?

@jadmsaadaot The ones that show up when we display a survey. In this case it's the next and previous button. This pr is made because there's some small hover artifacts since form io scss overrides some styles for the buttons.


Small fixes to How the payment ecosystem works

What's changed, or what was fixed?
In the How the payment ecosystem works article,

Fixed the capitalization of some link text that did not match the capitalization of the title of the article that is pointed to
Changed a period to a colon when introducing a list

CC: @petele

:tada: This has been pushed live to https://developers.google.com/web/


Fix bypass bug in filters

I think this PR fixes a bug in AudioKit.
The process function in the filters is ignoring the isPlaying property.
So if a filter had bypass enabled it would still process the frames
that were sent through it.
I considered refactoring to make a cleaner solution, but decided to do as little as possible to get this reviewed by you guys first. If this is a valid fix we can discuss options on how to implement the isPlaying-check in a better way. But that should probably be a separate PR so we can get this fix in production.
Keep up the good work!

Oh wow, thanks a lot!  I'm very open to changes to make the code more efficient or more readable.


Android Basics: Introduction to Room and Flow

I don't have a build.gradle project level file

In the android project view, it is listed as build.gradle (Project: Bus_Scheduler)

Thanks @tuxedotshirt ! That is a great way to find the project level build.gradle file. We'll be closing this issue since it's not an error with the codelab itself.


Write Tests for the Project

Possibly only unit tests using Mocha. Since integration tests would be taxing, and would be nice to have a little build system set up working good!

Frontend tests are kinda lame in my opinon. I think our best bet would to be to just write tests for the API. Tests can come from the admin panel, if we convert the admin panel to ES6, and angular 1.5


Is possible to get the current load via API?

We want to scale our DWU units based on load automatically, but I don't know how to to read current load values (number of queries, queued queries).
Does such API endpoint exist?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 95eb3903-d171-d3c9-180e-a49e14522f27
Version Independent ID: 44cece14-8e57-4d3b-6e5c-2a8ffe719be5
Content: Pause, resume, scale with REST APIs - Azure Synapse Analytics
Content Source: articles/synapse-analytics/sql-data-warehouse/sql-data-warehouse-manage-compute-rest-api.md
Service: synapse-analytics
Sub-service: sql-dw
GitHub Login: @Antvgski
Microsoft Alias: anvang

@himanshusinha-msft Thank you for your feedback . We will investigate and update the thread.

@MikhailTymchukFT Please refer to below article to monitor and see if that helps.

Monitor your Azure Synapse Analytics SQL pool workload using DMVs)

And below article to scale up/down:

Scale compute in Azure Synapse Analytics using T-SQL

Additional info:

Workload management concepts - Azure Synapse

@KranthiPakala-MSFT  Exactly what I need, thanks!


[okex-stream] all login-required channels become invalid after a network disconnection and reconnection

Steps to Reproduce:

Use the getUserTrades method to subscribe to order events.
Disconnect and then reconnect the network.
Place order on OKX; the order events will not be received.

@timmolter please check


Feature/publish selected object cloud

Added two services with corresponding back-end logic:

One to query the list of semantic instances in the map (return the list of instance_ids and their semantic category)
And one to query, given an instance_id, the aligned bounding box of that instance.

Also:

Refactored order of includes in a couple of files
Fixed semantic meshing which was ignoring a possible kFramesCountThresholdFactor
Refactored the extract_instances service to re-use code when retrieving the layer of a single instance.

Test PASSed.

Test PASSed.

Closing as contained in https://github.com/ethz-asl/voxblox-plusplus/pull/61


Feat/merge dev to stage

reference PR : https://github.com/zesty-io/website/pull/1275
Fix SSR issues
handle undefined window
replace mui-markdown with markdown-to-jsx
remove console logs
remove eslint warnings
wrap to options the markdown overrides
change npm run commit command to prevent re-asking of question.

@jomarmontuya @jeromeZestyIO
can you check these files cause we updated the mui-markdown to markdown-to-jsx(this will be compatible in ssr)
ContainerWithBackground.js
Features.js
NewBenifits.js
Cause we saw some css errors like this one , not showing the text


Can't update files after migration

Bug report

[x] I confirm this is a bug with Supabase, not with my own application.
[x] I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug
I can't update files in storage after migration from versions:
  studio:
    image: supabase/studio:20230216-e731b77
  kong:
    image: kong:2.8.1
  auth:
    image: supabase/gotrue:v2.47.0
  rest:
    image: postgrest/postgrest:v10.1.2
  realtime:
    image: supabase/realtime:v2.5.1
  storage:
    image: supabase/storage-api:v0.30.1
  imgproxy:
    image: darthsim/imgproxy:v3.13
  meta:
    image: supabase/postgres-meta:v0.60.7
  db:
    image: supabase/postgres:<IP_ADDRESS>-rc2

to versions:
studio:
    image: supabase/studio:20231123-64a766a
  kong:
    image: kong:2.8.1
  auth:
    image: supabase/gotrue:v2.125.1
  rest:
    image: postgrest/postgrest:v12.0.1
  realtime:
    image: supabase/realtime:v2.25.35
  storage:
    image: supabase/storage-api:v0.43.11
  imgproxy:
    image: darthsim/imgproxy:v3.8.0
  meta:
    image: supabase/postgres-meta:v0.75.0
  functions:
    image: supabase/edge-runtime:v1.29.1
  analytics:
    image: supabase/logflare:1.4.0
  db:
    image: supabase/postgres:<IP_ADDRESS>
  vector:
    image: timberio/vector:0.28.1-alpine

To Reproduce

Install the old verions.
Upload a new file to storage.
Upgrade to the new versions.
Upload a file with the same name on the same path.

Expected behavior
Overwrite the file without error.
Screenshots
The error appears in docker storage and is the following:
{"level":50,"time":"2024-01-16T08:23:19.589Z","pid":1,"hostname":"c1868c129091","error":{"raw":"{\"errno\":-20,\"code\":\"ENOTDIR\",\"syscall\":\"lstat\",\"path\":\"/var/lib/storage/stub/path/to/old/file/810562ed-e82c-4cdf-9e15-30d29dcef23f\"}","name":"Error","message":"ENOTDIR: not a directory, lstat '/var/lib/storage/stub/path/to/old/file/810562ed-e82c-4cdf-9e15-30d29dcef23f'","stack":"Error: ENOTDIR: not a directory, lstat '/var/lib/storage/stub/path/to/old/file/810562ed-e82c-4cdf-9e15-30d29dcef23f'"},"jodId":"","type":"event","event":"ObjectAdminDelete","payload":"{\"name\":\"file/path/to/old/file\",\"bucketId\":\"default\",\"tenant\":{\"ref\":\"stub\",\"host\":\"host-name\"},\"version\":\"810562ed-e82c-4cdf-9e15-30d29dcef23f\",\"reqId\":\"req-3dya\",\"$version\":\"v1\"}","objectPath":"path/to/old/file","tenantId":"stub","project":"stub","reqId":"req-3dya","msg":"[Admin]: ObjectAdminDelete path/to/old/file - FAILED"}
System information

OS: Ubuntu 22.04.3 LTS
Version of supabase-flutter: 2.0.2
Version of Flutter: 3.13.1

Additional context
I use the self-hosted version of Supabase.

Looking into this

@RichiB20 @GeoApopei18
There has been a change in the structure of how we store files on the filesystem and s3.
This change was introduced to improve the concurrency of the service.
To explain this further. In the older version which you used, we were storing files using the same file system structure as the file provided for example:
tenant-id/
├─ bucket/
   ├─ file.jpg

in the newer storage version, we are now nesting one level below with a random id:
tenant-id/
├─ bucket/
   ├─ file.jpg/
     ├─ 3a039063-742b-4e41-8177-e1142269ad78

since in Linux a folder name cannot co-exist with a file name with the same name (and vice versa) you are getting this issue.
The way you can go around this is to use the environment variable TUS_USE_FILE_VERSION_SEPARATOR=true
which will then store the files as follows:
tenant-id/
├─ bucket/
   ├─ file.jpg-$v-3a039063-742b-4e41-8177-e1142269ad78

and you'll be able to upgrade to the new version.
keep in mind this environment variable might be renamed in the future ( still keeping backwards compatibility )

@GeoApopei18 i don't think your issue is related
could you open a new issue please, i'll be happy to investigate.
Please provide the following information:
storage-version: from => to
storage-type: s3 / file
storage operation: upload / upload-upsert etc...
any other detail


Add extraEnv, extraVolumes, and extraVolumeMounts to prometheus-es-exporter

Thank you for this package - comes very useful in our projects!
This pull request adds a few parameters to the package that are useful to specify the exporter parameters, certificates and credentials stored in secrets and config maps.
For example, if elastic password and certificates are stored in secrets, it becomes possible to configure the exporter as follows via values.yaml:
image:
  tag: '0.14.0'
container:
  extraArgs:
  - '--basic-user=es-user'
  - '--basic-password=$(ES_PASSWORD)'
  - '--ca-certs=/mnt/certs/cacerts'
  extraEnv:
  - name: ES_PASSWORD
    valueFrom:
      secretKeyRef:
        name: es-credentials
        key: es-password
  extraVolumeMounts:
  - name: certs
    mountPath: /mnt/certs
pod:
  extraVolumes:
  - name: certs
    secret:
      secretName: es-credentials

I hope you will find it useful and merge.
I tried to follow the same conventions you established.
I also generated a new version of helm package tgz archive and regenerated index.yaml.

@braedon I also need this fix to use an ES TLS cert. Was just about to fork to add it. Is there a blocker to merging?

Apologies, I missed this. I'll take a look tomorrow.

Cool. Thanks. I'll also get another PR prepped for environment variable options so I can add basic auth support also.

Thanks for the PR @olegch, sorry it took me so long to get to it.
The additions look great. I'm going to merge into another branch so I can update the other chart as well, and update the minor version number. It should be in master in the next hour or so.

Version 0.2.0 should now be available in the helm repo with these changes.


kea: 1.9.7 -> 1.9.8

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/kea/versions.
meta.description for kea is: "High-performance, extensible DHCP server by ISC"
meta.homepage for kea is: "https://kea.isc.org/"
meta.changelog for kea is: ""
Updates performed

Version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-admin passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-ctrl-agent passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-dhcp-ddns passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-dhcp4 passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-dhcp6 passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-lfc passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/kea-shell passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/keactrl passed the binary check.

/nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin/perfdhcp passed the binary check.

9 of 9 passed binary check by having a zero exit code.

9 of 9 passed binary check by having the new version present in output.

found 1.9.8 with grep in /nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8

found 1.9.8 in filename of file in /nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8

directory tree listing: https://gist.github.com/262c91e373fc8d2c7152889bfdba5e5a

du listing: https://gist.github.com/26bda1a271c03eeca8d3cd3d399f7ce7

Rebuild report (if merged into master) (click to expand)

209 total rebuild path(s)

74 package rebuild(s)

74 x86_64-linux rebuild(s)
63 i686-linux rebuild(s)
14 x86_64-darwin rebuild(s)
58 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
alerta-server
appdaemon
armadillo
bpytop
cbonsai
cinnamon.cinnamon-common
cinnamon.cinnamon-control-center
cinnamon.cinnamon-gsettings-overrides
cinnamon.cinnamon-screensaver
cinnamon.cinnamon-session
cinnamon.cinnamon-settings-daemon
cinnamon.cinnamon-translations
cinnamon.nemo
citrix_workspace
citrix_workspace_20_04_0
citrix_workspace_20_06_0
citrix_workspace_20_09_0
citrix_workspace_20_10_0
citrix_workspace_20_12_0
citrix_workspace_21_01_0
citrix_workspace_21_03_0
cura
diffoscope
dnsname-cni
elinks
exploitdb
freeoffice
gdbgui
gns3-gui
gns3-server
gnss-sdr
home-assistant
kea
libredirect
moodle-dl
nextcloud-client
nixos-install-tools
octoprint
onionshare
onionshare-gui
openmolcas
python38Packages.aioambient
python38Packages.cassandra-driver
python38Packages.flask-socketio
python38Packages.python-engineio
python38Packages.python-engineio_3
python38Packages.python-socketio
python38Packages.python-socketio_4
python38Packages.r2pipe
python38Packages.sentry-sdk

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A kea https://github.com/r-ryantm/nixpkgs/archive/82bc990cdf8fc5d8247d5fce80c031d28323e933.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8
ls -la /nix/store/6wq02mhsspi9f7maxrvzcig6gp7g0d79-kea-1.9.8/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

kea

Maintainer pings
cc @fpletz for testing.

Result of nixpkgs-review pr 125131 at 82bc990c run on x86_64-linux 1

1 package built successfully:

kea

1 suggestion:

warning: missing-patch-comment
Consider adding a comment explaining the purpose of this patch on the line preceeding.
Near pkgs/tools/networking/kea/default.nix:22:15:
   |
22 |   patches = [ ./dont-create-var.patch ];
   |               ^


Update node.rb

There is a required field on linux.

Thanks for the pull request. Released 5.0.1.


Issue: Update ToolsPath to Use DirectoryPath

Description: The ToolsPath property in the Database class currently has a hardcoded path. This should be updated to dynamically combine DirectoryPath with "Tools" using Path.Combine for better flexibility.
Current code:
https://github.com/mihailfox/XivVoices/blob/f4933c896afa925d58feade6e9030bab8dee1ccf/XivVoices/Engine/Database.cs#L31
Proposed change:
public string ToolsPath { get { return Path.Combine(DirectoryPath, "Tools"); } }

Please delete this repo and fork the new one by the new dev who continued the development of this project:
https://github.com/FantasticalMouthpiece/XivVoices


Negative Variances from predictive_gradients()

I'm trying to get at the gradient of the predictions, but the variance of the estimate is coming back negative.  Not sure if what I'm doing is wrong, but as a MWE, the following seems to demonstrate the issue:
import GPy
import numpy as np

# keep deviates deterministic (actual seed not important)
np.random.seed(123)

# draw from our function
X = np.random.uniform(-3.,3.,(20,1))
Y = np.sin(X) + np.random.randn(20,1)*0.05

# set up model
m = GPy.models.GPRegression(X, Y)

# optimise parameters (optional)
m.optimize()

# where do we want to predict our function
Xp = np.linspace(-4,4,51)[:,None]

# get our predictions of mean and gradient
mu,var = m.predict(Xp)
mug,varg = m.predictive_gradients(Xp)

# display variance in estimate of gradient?
print(varg)

I get back a number of negative values for the variance, which I'm not expecting…  If I plot my test points and the function being fit they look fine:

But the variance seems to have a lot of unexpected structure, i.e. doesn't just look like numerical instability to me.  Not sure if this matters, but I'm running without Weave (I'm under OS X and am currently struggling to get OMP code compiling) so don't know if the C version is correct.
I've just started to use GPy, so sorry if this issue/question is malformed!  While poking around I've just seen m._raw_predict() which looks useful as well.  The underscore prefix makes it look as though it should be internal/private, is it recommended to be using this function?

Just tried under Linux with latest versions (Python=2.7.9, numpy=1.9.2, scipy=0.15.1, GPy=0.6.0) and I get the same result, so probably nothing to do with the lack of working Weave under OS X.

As far as I know predictive gradients actually returns the derivative of the mean and variance predictions, not the predictions of the derivative of the GP. So what you're seeing is that the variance of your function is decreasing between -4 and -1, is roughly constant up to 3 and then starts increasing between 3 and 4.

OK, thanks for the note.
AFAICT, the derivative of the mean should be the same as the derivative of the latent function—at least for the RBF kernel that I care about!  Not that this matters for me, but I presume this holds for most covariance functions?
I'm still unsure how to get at the variance of an estimate of the derivative of the latent function using the framework provided by GPy. I'll continue to read around the subject, but any pointers would be greatly appreciated!

The derivative of the mean should be the same for all kernel functions.
So I actually had to implement that myself for RBF functions. Couldn't figure out an easy generic way for all kernels either, mostly because there doesn't seem to be a way to get the second derivative of the kernel, which is needed for the prior.
Anyways, since your just interested in RBFs anyways, here's my rbf-specific code....without any warranty or me saying that this is a good implementation in any way. There might be some way that is a lot better, I'm not involved in this project at all, just a user. If you find a bug please let me know.
def predict_derivatives(self, x_new):
    """
    Predict derivatives of a gp at the points x_new.

    Author: Felix Berkenkamp (befelix)

    Parameters:
    -----------derivative
    self: instance of GPy.core.gp
        With RBF kernel function
    x_new: 2d-array
        Every row is a new data point at which to evaluate the derivative
derivative
    Returns:
    --------
    mu: 2d-array
        The mean derivative
    var: 2/3d-array
        If there is only one data point to predict var is a 2d array with
        the variance matrix. Otherwise it is a 3d matrix where
        var[i, :, :] is the ith covariance matrix.
    """
    if not self.kern.name == 'rbf':
        raise AttributeError('The variance prior only works for RBF kernels.')
    x_new = np.atleast_2d(x_new)

    # Compute mean, initialize variancederivative
    mu = self.kern.gradients_X(self.posterior.woodbury_vector.T, x_new, self.X)
    var = np.empty((x_new.shape[0], x_new.shape[1], x_new.shape[1]),
                   dtype=mu.dtype)

    # Make sure lengthscales are of the right dimensions
    lengthscales = self.kern.lengthscale.values
    if not lengthscales.shape[0] == self.X.shape[1]:
        lengthscales = np.tile(lengthscales, (self.X.shape[1],))

    def dk_dx(X, X2):
        """Compute the derivative of k(X, X2) with respect to X."""

        # Derivative with respect to r
        dK_dr = self.kern.dK_dr_via_X(X, X2)

        # Temporary stuff
        tmp = self.kern._inv_dist(X, X2) * dK_dr

        # dK_dX = dK_dr * dr_dx
        # dr_dx1 = invdist * (x1 - x'1) / l1**2
        dk_dx = np.empty((X2.shape[0], X.shape[1]), dtype=np.float64)
        for q in range(self.input_dim):
            dk_dx[:, q] = tmp * (X[:, q, None] - X2[None, :, q])
        return dk_dx / (lengthscales ** 2)

    # Compute derivative variance for each test point
    for i in range(x_new.shape[0]):
        dk = dk_dx(x_new[None, i, :], self.X)
        # Would be great if there was a way to get the prior directly from the
        # library
        # But I think only d1 is implemented
        var[i, :, :] = np.diag(self.kern.variance / (lengthscales ** 2)) -\
            np.dot(dk.T, np.dot(self.posterior.woodbury_inv, dk))

    # If there was only one test point to begin with, squeeze the
    # corresponding dimension
    if x_new.shape[0] <= 1:
        var = var.squeeze(0)
    return mu, var

Wow, thanks!
I've got few different blobs of code to do the same thing now, but this is the first that has any integration into GPy.  I'll spend some time making sure they are consistent, should help with debugging as well as my understanding.
Have you asked on the gpy-users mailing list about ways to approach this?

Wasn't even aware there was an active user mailing list, guess I'll subscribe to that :)
If you figure out a better/faster way to do this please let me know!

Yes, have a look at:
https://lists.shef.ac.uk/sympa/info/gpy-users
I've just had a play with your code and it's consistent with my numerical estimates…  Will play more tomorrow.

This bug arose as a misnderstanding of the predictive_gradients function. Have opened a new bug to request a function with the desired behaviour. #213


[freeglut] enable building on non-win32

port patch from #5169

Hi @cenit
We have a green check, but it is a false positive. For some reason CI didn't rebuild freeglut for Mac OS, I'll delete the cached archive and trigger a new build for it.

Thanks for the PR!
freeglut now works on x64-osx

x64-osx
master
test
notes

gl2ps
Skip
Pass

freeglut
Fail
Pass

theia
Skip
Fail

I hope that theia will be fixed in its own separate PR soon :)


[UWP] Add support for action overflow

Related Issue
Fixes #5494
Fixes #5495
Description
Add support for action overflow to UWP. The majority of the code in this PR is in aid of supporting the spec'd show card behavior where a show card in an overflow menu needs to switch places with the last primary button when invoked. This required more logic and bookkeeping around show card actions.
As part of that bookkeeping, more information needed to be registered with the render context via AddInlineShowCard. Because of that, that function started to feel more implementation specific than felt right for the public API surface. With this change, hosts who are custom rendering ActionSets (not individual actions) will need to manage their own inline show cards. This seems like a reasonable requirement for a host custom rendering the entire action set.
Sample Card
Moved the existing sample card to the Test directory and added some more cases to it. Created a new sample card in elements to be the sample for the webpage. With this change the two sample cards for overflow are:
\samples\v1.5\Elements\Action.Mode.json
samples\v1.5\Test\ActionModeTestCard.json
How Verified

Updated and verified behavior in the two sample cards discussed above
Updates to the object model unit tests to verify new properties
Renderer test to validate OverflowMaxActions behavior

Microsoft Reviewers: Open in CodeFlow

/* Header file automatically generated from AdaptiveCards.ObjectModel.Uwp.idl */

Should this file be added? It looks like it's a product of some compilation

Refers to: source/uwp/AdaptiveCardsObjectModel/AdaptiveCards.ObjectModel.Uwp.h:1 in cd79747. [](commit_id = cd797472b9c6f1508495df11ba4c9e738049ffab, deletion_comment = False)

/* Header file automatically generated from AdaptiveCards.ObjectModel.Uwp.idl */

It should not. Unfortunately it was accidently added to this branch before it was added to .gitignore in another change list. Now i can't get git to notice I'm deleting it to remove it from this change list even if i take it out of the .gitignore. I might have to delete it in a separate change to un-confuse git.

In reply to: 892048249

Refers to: source/uwp/AdaptiveCardsObjectModel/AdaptiveCards.ObjectModel.Uwp.h:1 in cd79747. [](commit_id = cd797472b9c6f1508495df11ba4c9e738049ffab, deletion_comment = False)


Prepare open source

Closes: #8
Description
Add guidelines documents for the open sourcing of this repository.

For contributor use:

[ ] Targeted PR against master branch
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work.
[ ] Code follows the standards mentioned here.
[ ] Updated relevant documentation
[ ] Re-reviewed Files changed in the Github PR explorer
[ ] Added appropriate labels

@gus-labs files look good, did you do the stuff on this list? https://github.com/onflow/fcl-contracts/issues/8
Branch rules and permissions, I checked the ones I'm aware that are done

@gus-labs files look good, did you do the stuff on this list? #8
Branch rules and permissions, I checked the ones I'm aware that are done

Yes, I did. I was given repo admin rights two days ago.

Yes, I did. I was given repo admin rights two days ago.

awesome, then I believe we are good to go!


Visibility threshold for pie charts

What does this PR do? How does it affect users?
This PR adds a visibilityThreshold option to chartOptions for pie and donut charts. Any percentage that falls below the visibility threshold will be placed in an "Other" slice of the pie chart.
Major changes:

Adds logic in lib/libraries/index.js#136 to check to visibility threshold and calculate what items should go in the Other category.
Moves the pie chart nightwatch test examples to a separate html file so we can test pie charts with different options. Nightwatch tests also run slightly faster because the new html file loads faster than test/demo/index.html.

Before

After

How should this be tested?
Step through the code line by line. Things to keep in mind as you review:

Are there any edge cases not covered by this code?
Does this code follow conventions (naming, formatting, modularization, etc) where applicable?

Fetch the branch and/or deploy to staging to test the following:

[ ] Does the code compile without warnings (check shell, console)?
[ ] Do all tests pass?
[ ] Does the UI, pixel by pixel, look exactly as expected (check various screen sizes, including mobile)?
[ ] If the feature makes requests from the browser, inspect them in the Web Inspector. Do they look as expected (parameters, headers, etc)?
[ ] If the feature sends data to Keen, is the data visible in the project if you run an extraction (include link to collection/query)?
[ ] If the feature saves data to a database, can you confirm the data is indeed created in the database?

Related tickets
https://groups.google.com/a/keen.io/forum/#!topic/dashboards/9i0_8SNIKNo

Oops, the correct PR is here: https://github.com/keen/keen-dataviz.js/pull/43


Failure to warn about undefined variable

Environment data

VS Code version: 1.39.1
Extension version (available under the Extensions sidebar): 2019.10.41019
OS and version: macOS 10.14.5
Python version (& distribution if applicable, e.g. Anaconda): 3.7 (MacOS 64-bit installer https://www.python.org/downloads/release/python-375/)
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): venv
Relevant/affected Python packages and their versions: n/a
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info microsoft/vscode-python#3977): Not set; Microsoft Python Language Server is booting up as shown in output window

Expected behaviour
when test and frog are undefined, the line of code
test = frog

produces a warning: Undefined variable: 'frog'.  This is the desired behavior.
When test is undefined, the line of code
test = test

should also produce an undefined variable warning in the warning window.
Actual behaviour
In the second example, vscode does not recognize that test is undefined.
Steps to reproduce:

create empty file with .py extension
select python virtual environment with python 3.7
add test = test to file
no warning will appear

Logs
Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)
> conda --version
> pyenv root
> python3.7 -c "import sys;print(sys.executable)"
> python3.6 -c "import sys;print(sys.executable)"
> python3 -c "import sys;print(sys.executable)"
> python2 -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> python -c "import sys;print(sys.executable)"
> conda info --json
> /Library/Frameworks/Python.framework/Versions/3.7/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.7/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.7/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/local/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /usr/bin/python2.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
><EMAIL_ADDRESS>~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
><EMAIL_ADDRESS>~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
><EMAIL_ADDRESS>~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
><EMAIL_ADDRESS>~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
><EMAIL_ADDRESS>~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
><EMAIL_ADDRESS>~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
Starting Microsoft Python language server.
> conda info --json
> conda --version
> ~/Downloads/no-undefined-mwe/env/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3.7 ~/.vscode/extensions/ms-python.python-2019.10.41019/pythonFiles/interpreterInfo.py
> ~/Downloads/no-undefined-mwe/env/bin/python3.7 -m pip --version

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)
XXX

Our undefined variable checker runs after the analysis and sees that test was defined and doesn't warn on it. There are some conditions where we don't warn about forward references (since it's often difficult to reason about if some value may or may not have been assigned yet), and this is one of them.
Feasibly we could detect this case, but we'd have to shove the undefined checks backwards into the analysis, and even then it might not work at the moment due to our symbol table likely containing test as a variable yet-to-be processed.

Thank you for taking the time to review the issue and to explain what is happening.  Based on my experience working with other interpreters, this feature would seem to be an expectation of many developers.
In a typescript file in Visual Studio Code with the following line, you would get a warning if test were undefined:
const test: any = test;

In a java file in IntelliJ IDEA with the following line, you would get a warning if test were undefined:
int test = test;

As a developer, I would expect this check to happen, but I will defer to the maintainers of this project as to how much priority it deserves.  Thank you again for getting back to me.

You're right, just know that it's a bit more complicated for a dynamic language like Python. An example where this sort of thing gets tricky is:
import random

def set_test(v):
    global test
    test = v

if random.choice((True, False)):
    set_test(123)

test = test

In another language, you'd be forced to define test somewhere; in Python you can defer that sort of thing until later. We try to be as helpful as possible in terms of completion and will assume that there is a variable test in the global scope if test = ... happened. If that turns out to not be the case at runtime, then it'd break.
Now, this whole thing can be instead compared to an analyzer being able to prove that something was definitely assigned or not (like some static analyzers that know when you're about to incorrectly use null), which is possible, but still non-trivial.

I hear you, and to be honest I can appreciate the huge amount of work that goes into maintaining a project like this and know that I shouldn't expect the same amount of safety when using a dynamic language.
Getting a warning when a variable was not definitely assigned would for sure be a nice feature, and I hope that you'll consider it, but I definitely don't expect it to happen overnight.

Sure thing, I just wanted to write down my thoughts while I was looking at this issue. 🙂


Permanent Virtual Machines

It shall be possible to mark a virtual machine as permanent with a label (e.g. type=permanent, permanent=true, lifetime=permanent etc.). This special label shall be assigned when the virtual machine is created (using either create-instance or create-group) based on a label/tag that is set in the groups file:
name: teachers
users:
- email<EMAIL_ADDRESS>  ssh-key: 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF4JGW5PzpeGh1oFyultAos1RUeJSpexOaQ8odLKmVWV<EMAIL_ADDRESS>  name: patrick_bucher
  labels:
    - permanent: true

The commands destroy-instance and destroy-group must not delete virtual machines marked as permanent, unless an additional switch (e.g. --include-permanent) is set. The stop-instance and stop-groups commands remain the same, for permanent virtual machines must still be shutdown after usage. The start-instance and start-group commands remain the same, too.

The concept of labels shall not be exposed to the configuration layer. It's an implementation detail. Therefore, the permanent flag shall not be a label, but an additional setting for each user:
name: teachers
users:
  - email<EMAIL_ADDRESS>    name: patrick_bucher
    ssh-key: "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJhx+5Z/Ao+9w3EujyXcKcE8Wzw0uyApawro5dlMXQwa<EMAIL_ADDRESS>    permanent: true

Resolved in v0.0.7.


Ignore base image layer

This PR makes changes to ignore the base layer from getting pulled or calculated as part of the size. (Since the Windows base layer is no longer a foreign layer, it current gets pulled and extracted into every package directory.) It adds a check to ignore the first layer if there is more than one layer in the manifest. (Another option would be to only pull the last layer, but I figured ignoring the first layer is "safer.")

Closes #97


Show if a line is passing a character limit

Is there any way in VSCode to show/highlight characters passing a character limit for a line such as 80 (possibly ignoring whitespace in the left side)? I sometimes use Vim, in Vim, there is a script for that. I have searched the VSCode Marketplace and have found nothing. It would be useful if a feature like this was pre-implemented in VSCode.

I think you want "Editor: Rulers" in settings:

If you specifically want highlighting, an extension could implement this via the text decorations API, here's a ref to get started: https://code.visualstudio.com/api/references/vscode-api#window.createTextEditorDecorationType. We're not likely to implement this in core as we already have the rulers.


Support arm64 architecture

Comment:
This seems to have better architecture support than the conda julia package, however it still missed linux arm64.
https://anaconda.org/conda-forge/juliaup

linux-64
osx-arm64
windows-64
osx-64

This one is probably easier to do than julia itself.


Menu contributions to a menu that doesn't exist where the when is false shouldn't show an error

See https://github.com/microsoft/vscode-pull-request-github/issues/3073 for a specific example.
Repro:

Create an extension that contributes a command to a menu that doesn't exist.
Add a when to that contribution that evaluates to false.
Run the extension and see an error " is not a valid menu identifier".

Since the when is false, I would not expect to see that error.

The error also shows up even if you don't have a when clause right?

What is the use case here?
I believe the current behavior is to show a warning, not an error, which makes sense for the author of that extension. Would you suggest to just print to devtools?

For GitHub Pull Requests and Issues, we want to contribute a "Create PR" command to a menu the Remote Repositories extension adds. When the Remote Repositories extension isn't installed, we have this warning surfacing in several places:

Ideally, this wouldn't show anywhere as it makes it looks like something is wrong when in reality there's no problem.

I didn't really test it... but does the warning actually pop up in the UI, or is it only a devtools thing?

It shows in the Running Extensions editor.

I think this is fine for two reasons:

There's no user notification popup. It actually appears in the Running Extensions editor, a UI piece mainly targetting extension authors, who happen to be the right people that should see the message.
It's a warning, not an error

I suggest we just close it.
I'm wondering, though, how did @EugeneGohh actually find it out and filled an issue about it?

Can you show me a screenshot, as I can't reproduce this?

@joaomoreno hope this helps

That notification is unrelated to the warning.


feat: :sparkles: added steam_workshop_id to ModManifest

Added an optional field steam_workshop_id to ModManifest.
In the editor, it is validated with a basic string length check. Once loaded from a workshop path, it's validated based on the zip_path. If invalid, it's overridden by the correct ID retrieved from the zip_path.
closes #390

This PR seems to introduce a parser error at path.gd:58 as "zip_path_global" is not declared


Login Button Color

Is it a ?

[x] Feature Request

Feature Request
Feature Description
Log In button color should be same a sign-up color i.e ( shade of green )

We have deliberately made it so


Command history, Command not found - future improvements

possibility to enable and use notification about command not found
#ifdef COMMAND_NOT_FOUND_MESSAGE
	if (!success) {
	   this->serial.println(COMMAND_NOT_FOUND_MESSAGE);
	}
#endif

useful for command history implementation(like in Linux console):
(for last 5 or 10 commands)
#define KEYCODE_UP 65
#define KEYCODE_DOWN 66
/**************************/
case KEYCODE_UP:
  if (length > 0) {
    // Move pointer
    length = 0;
    this->serial.write("\33[2K\r");
  }
break;
case KEYCODE_DOWN:
  if (length > 0) {
    // Move pointer
    length = 0;
    this->serial.write("\33[2K\r");
}
break;

and this future must have possibility to be enabled like plugin from #define

Hi,
Thanks for the feedback. I will look into this when I have the time :-)
Thanks!

This is added in version v2.1.0 that I just pushed.
I have added basic support for history (up/down only, no inline editing or preserving original input) and added generic pre/post callbacks to handle something like command not found.


License and copyright

Hi Roman,
for using your work as my personal website and respecting your work, need i to add some credits to you in the website? Of course i have added license in main working tree.
Thanks! Giorgio

Hey there! The code is all Apache 2 licensed, but attribution can be kept in the code, no need to add anything to the website itself. Thanks!


XWayland crashes, does not recover, requires Hyprland reboot

Bug or Regression?
Bug
System Info and Version
 System/Version info

Hyprland, built from branch  at commit 9e781040d9067c2711ec2e9f5b47b76ef70762b3  (props: bump version to 0.41.1).
Date: Thu Jun 13 09:54:06 2024
Tag: v0.41.1, commits: 4818

flags: (if any)

System Information:
System name: Linux
Node name: Callisto
Release: 6.9.4-arch1-1
Version: #1 SMP PREEMPT_DYNAMIC Wed, 12 Jun 2024 20:17:17 +0000

GPU information:
c1:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Phoenix1 [1002:15bf] (rev c2) (prog-if 00 [VGA controller])

os-release: NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://gitlab.archlinux.org/groups/archlinux/-/issues"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

plugins:

======Config-Start======

Config File: /home/moonleay/.config/hypr/hyprland.conf: Read Succeeded
# moonleay's Hyprland config

# Source other config files
source = ~/.config/hypr/monitors.conf # Configure monitors
source = ~/.config/hypr/env_vars.conf # Set environment variables
source = ~/.config/hypr/autorun.conf # Autorun apps
source = ~/.config/hypr/input.conf # Setup inputs
source = ~/.config/hypr/keybindings.conf # Setup keybindings
source = ~/.config/hypr/windowrules.conf # Setup windowrules
source = ~/.config/hypr/basics.conf # Setup Hyprland basics (layout, design, etc.)
# source = ~/.config/hypr/mocha.conf # Setup Colors

# Other configs:

# Disable xwayland scaling
xwayland {
  force_zero_scaling = true
}

misc {
    disable_hyprland_logo = true
    # lower framerate when not updating screen (powersaving)
    vfr = true
}

debug {
    disable_logs = false
}

# Per device configs
# <none>

Config File: /home/moonleay/.config/hypr/monitors.conf: Read Succeeded
# Monitors
# monitor=,preferred,auto,auto
# monitor=eDP-2,preferred,0x0,2
monitor=eDP-1,2560x1600@160,0x0,1.6 # 1.6
monitor=DP-2,3840x2160@120,1600x0,1.25 # 1.2
# monitor=HDMI-A-1,preferred,4480x0,1
# monitor=HDMI-A-1,3440x1440@100,1280x0,1

Config File: /home/moonleay/.config/hypr/env_vars.conf: Read Succeeded
# Environment variables
env = MOZ_ENABLE_WAYLAND,1
env = MOZ_DISABLE_RDD_SANDBOX,1
env = WLR_DRM_NO_ATOMIC,1
env = GDK_BACKEND,wayland,x11
env = QT_QPA_PLATFORM,wayland;xcb
env = SDL_VIDEODRIVER,wayland
env = CLUTTER_BACKEND,wayland
env = XDG_CURRENT_DESKTOP,Hyprland
env = XDG_SESSION_TYPE,wayland
env = XDG_SESSION_DESKTOP,Hyprland

env = QT_AUTO_SCREEN_SCALE_FACTOR,1
env = QT_QPA_PLATFORM,wayland;xcb
env = QT_WAYLAND_DISABLE_WINDOWDECORATION,1
env = QT_QPA_PLATFORMTHEME,qt5ct

env = HYPRCURSOR_THEME,adwaita
env = HYPRCURSOR_SIZE,24
env = XCURSOR_THEME,adwaita
env = XCURSOR_SIZE,24
# env = ENABLE_DEVICE_CHOOSER_LAYER=1 VULKAN_DEVICE_INDEX=0

Config File: /home/moonleay/.config/hypr/autorun.conf: Read Succeeded
# Autorun apps
exec-once = dunst & hyprpaper & waybar
exec-once = dbus-update-activation-environment --systemd WAYLAND_DISPLAY XDG_CURRENT_DESKTOP
exec-once = hyprctl setcursor Adwaita 24
exec-once = gsettings set org.gnome.desktop.interface cursor-theme 'adwaita'
exec-once = gsettings set org.gnome.desktop.interface cursor-size '24'
exec-once = bash /home/moonleay/.config/hypr/hyprland/scripts/fixXwaylandVideoBridge.sh
exec-once = solaar -w hide
exec-once = setxkbmap -model 'pc105angle' -layout 'us' -variant 'cmk_ed_us' -option'' -option 'misc:extend,lv5:caps_switch_lock,grp:shifts_toggle,compose:menu,misc:cmk_curl_dh'
exec-once = discord & librewolf
# exec-once = flatpak run org.kde.xwaylandvideobridge
# exec-once = gBar bar 0 & gBar bar 1 & gBar bar 2
# exec-once = /usr/lib/polkit-kde-authentication-agent-1

Config File: /home/moonleay/.config/hypr/input.conf: Read Succeeded
# Input settings
input {
    kb_layout = us
    kb_model = pc105angle
    # kb_variant = colemak_dh
    kb_variant = cmk_ed_us
    kb_options = misc:extend, lv5:caps_switch_lock, grp:shifts_toggle, compose:menu, misc:cmk_curl_dh

    repeat_rate = 60
    repeat_delay = 200

    kb_file =
    kb_rules = evdev

    follow_mouse = 1

    touchpad {
	disable_while_typing = yes
        natural_scroll = yes
    }

    sensitivity = 0 # -0.2 #0 # -1.0 - 1.0, 0 means no modification.
    accel_profile = flat
}

Config File: /home/moonleay/.config/hypr/keybindings.conf: Read Succeeded
# Keymappings
$mainMod = SUPER
$secondaryMod = Alt_L

# Custom mappings

# Shutdown / Reboot
bind = $secondaryMod, XF86Eject, exec, shutdown now
bind = $secondaryMod SHIFT, XF86Eject, exec, reboot
bind = $secondaryMod, XF86Launch1, exec, shutdown now
bind = $secondaryMod SHIFT, XF86Launch1, exec, reboot

# Music control
bind = , XF86Tools, exec, playerctl previous
bind = , XF86Launch5, exec, playerctl play-pause
bind = , XF86Launch6, exec, playerctl next

# Run "overlay apps"
bind = CTRL ALT, DELETE, exec, swaylock
bind = $secondaryMod SHIFT, s, exec, grim -g "$(slurp)" -c "/home/moonleay/Pictures/Screenshots/$(date +'%s_grim.png')"
bind = $secondaryMod Control_L, s, exec, grim -c "/home/moonleay/Pictures/Screenshots/$(date +'%s_grim_full.png')"

# Run application
bind = $secondaryMod, q, exec, librewolf

# Force app fullsceen
bind = ALT_L, f, fullscreen
bind = ALT_L SHIFT, f, fakefullscreen

# Custom rofi
bind = $secondaryMod, TAB, exec, rofi -show window -show-icons

# Exit applications
bind = $mainMod, C, killactive,
bind = $secondaryMod, C, killactive,
bind = $secondaryMod, M, exit,

# Run applications
bind = $secondaryMod, T, exec, alacritty
bind = $secondaryMod, E, exec, thunar

# Control open applications
bind = $secondaryMod, V, togglefloating,
bind = $mainMod, V, togglefloating,
bind = $secondaryMod, R, exec, rofi -show drun -show-icons
bind = $secondaryMod, P, pseudo, # dwindle
bind = $secondaryMod, J, togglesplit, # dwindle

# Move the window focus
bind = $mainMod, left, movefocus, l
bind = $mainMod, right, movefocus, r
bind = $mainMod, up, movefocus, u
bind = $mainMod, down, movefocus, d

# Switch focus
bind = $mainMod, 1, workspace, 1
bind = $mainMod, 2, workspace, 2
bind = $mainMod, 3, workspace, 3
bind = $mainMod, 4, workspace, 4
bind = $mainMod, 5, workspace, 5
bind = $mainMod, 6, workspace, 6
bind = $mainMod, 7, workspace, 7
bind = $mainMod, 8, workspace, 8
bind = $mainMod, 9, workspace, 9
bind = $mainMod, 0, workspace, 10
bind = $mainMod, minus, workspace, 11
bind = $mainMod, equal, workspace, 12

# Move window to other workspace
bind = $mainMod SHIFT, 1, movetoworkspace, 1
bind = $mainMod SHIFT, 2, movetoworkspace, 2
bind = $mainMod SHIFT, 3, movetoworkspace, 3
bind = $mainMod SHIFT, 4, movetoworkspace, 4
bind = $mainMod SHIFT, 5, movetoworkspace, 5
bind = $mainMod SHIFT, 6, movetoworkspace, 6
bind = $mainMod SHIFT, 7, movetoworkspace, 7
bind = $mainMod SHIFT, 8, movetoworkspace, 8
bind = $mainMod SHIFT, 9, movetoworkspace, 9
bind = $mainMod SHIFT, 0, movetoworkspace, 10
bind = $mainMod SHIFT, minus, movetoworkspace, 11
bind = $mainMod SHIFT, equal, movetoworkspace, 12

# Move window to other workspace, but silent
bind = $mainMod Control, 1, movetoworkspacesilent, 1
bind = $mainMod Control, 2, movetoworkspacesilent, 2
bind = $mainMod Control, 3, movetoworkspacesilent, 3
bind = $mainMod Control, 4, movetoworkspacesilent, 4
bind = $mainMod Control, 5, movetoworkspacesilent, 5
bind = $mainMod Control, 6, movetoworkspacesilent, 6
bind = $mainMod Control, 7, movetoworkspacesilent, 7
bind = $mainMod Control, 8, movetoworkspacesilent, 8
bind = $mainMod Control, 9, movetoworkspacesilent, 9
bind = $mainMod Control, 0, movetoworkspacesilent, 10
bind = $mainMod Control, minus, movetoworkspacesilent, 11
bind = $mainMod Control, equal, movetoworkspacesilent, 12

# Scroll through workspaces
bind = $mainMod, mouse_down, workspace, e+1
bind = $mainMod, mouse_up, workspace, e-1

# extend specific
bind = MOD3SUPER, n, movefocus, l
bind = MOD3SUPER, i, movefocus, r
bind = MOD3SUPER, u, movefocus, u
bind = MOD3SUPER, e, movefocus, d

bind = MOD3SUPERCTRL, l, togglespecialworkspace
bind = MOD3SUPERCTRL, y, focuscurrentorlast
bind = MOD3SUPERCTRL, i, workspace, r+1
bind = MOD3SUPERCTRL, n, workspace, r-1

bind = MOD3SUPERCTRLSHIFT, l, movetoworkspace, 1
bind = MOD3SUPERCTRLSHIFt, y, movetoworkspace, e+1
bind = MOD3SUPERCTRLSHIFT, i, movetoworkspace, +1
bind = MOD3SUPERCTRLSHIFT, n, movetoworkspace, -1

# Move and scale workspaces
bindm = $mainMod, mouse:272, movewindow
bindm = $mainMod, mouse:273, resizewindow

bindm = $secondaryMod, mouse:272, movewindow
bindm = $secondaryMod, mouse:273, resizewindow

Config File: /home/moonleay/.config/hypr/windowrules.conf: Read Succeeded
# Windowrules
windowrulev2 = immediate, class:^(minecraft)$
windowrulev2 = workspace 2, monitor 1, class:^(LibreWolf)$
windowrulev2 = workspace 1, monitor 1, class:^(discord)$

# Screensharing
windowrulev2 = opacity 0.0 override 0.0 override,class:^(xwaylandvideobridge)$
windowrulev2 = noanim,class:^(xwaylandvideobridge)$
windowrulev2 = nofocus,class:^(xwaylandvideobridge)$
windowrulev2 = noinitialfocus,class:^(xwaylandvideobridge)$

# -- Fix odd behaviors in IntelliJ IDEs --
#! Fix focus issues when dialogs are opened or closed
windowrulev2 = windowdance,class:^(jetbrains-.*)$,floating:1
#! Fix splash screen showing in weird places and prevent annoying focus takeovers
windowrulev2 = center,class:^(jetbrains-.*)$,title:^(splash)$,floating:1
windowrulev2 = nofocus,class:^(jetbrains-.*)$,title:^(splash)$,floating:1
windowrulev2 = noborder,class:^(jetbrains-.*)$,title:^(splash)$,floating:1

#! Center popups/find windows
windowrulev2 = center,class:^(jetbrains-.*)$,title:^( )$,floating:1
windowrulev2 = stayfocused,class:^(jetbrains-.*)$,title:^( )$,floating:1
windowrulev2 = noborder,class:^(jetbrains-.*)$,title:^( )$,floating:1
#! Disable window flicker when autocomplete or tooltips appear
windowrulev2 = nofocus,class:^(jetbrains-.*)$,title:^(win.*)$,floating:1

# windowrulev2 = suppressevent maximize, class:.*

Config File: /home/moonleay/.config/hypr/basics.conf: Read Succeeded
# General Hyprland settings
general {
    gaps_in = 5
    gaps_out = 10 #20
    border_size = 2
    col.active_border = rgba(33ccffee) rgba(00ff99ee) 45deg
    col.inactive_border = rgba(595959aa)

    layout = dwindle
    allow_tearing = true
}

# Window decorations
decoration {
    rounding = 5

    blur {
    	# Disable blur for battery saving
        enabled = false
        size = 3
        passes = 1
    }

    drop_shadow = yes
    shadow_range = 4
    shadow_render_power = 3
    col.shadow = rgba(1a1a1aee)
}

# Window animations
animations {
    enabled = yes

    bezier = myBezier, 0.05, 0.9, 0.1, 1.05

    animation = windows, 1, 7, myBezier
    animation = windowsOut, 1, 7, default, popin 80%
    animation = border, 1, 10, default
    animation = borderangle, 1, 8, default
    animation = fade, 1, 7, default
    animation = workspaces, 1, 6, default
}

# Dwindle layout
dwindle {
    pseudotile = yes
    preserve_split = yes
}

master {
    new_is_master = true
}

gestures {
    workspace_swipe = true
    workspace_swipe_forever = true
    workspace_swipe_min_speed_to_force = 5
    workspace_swipe_distance = 400
    workspace_swipe_direction_lock = false
}

======Config-End========

Description
Sometimes when I am using JetBrains IDEs, xwayland crashes completely and does not recover until Hyprland is relaunched.
How to reproduce
I have no clue, but it happened 3 times now.
But I suspect it may be caused by running java applications, since I always encountered the issue when I was using a JB IDE.
Crash reports, logs, images, videos
hyprlog.log
There is a xwayland / xwm crash in line 148712.

[LOG] Window 633abf0ff260 set title to xwayland crash does not open back up - Kagi Search — LibreWolf
[LOG] Searching for matching rules for LibreWolf (title: xwayland crash does not open back up - Kagi Search — LibreWolf)
[LOG] Window rule workspace 2 ->  monitor 1, class:^(LibreWolf)$ matched [Window 633abf0ff260: title: "xwayland crash does not open back up - Kagi Search — LibreWolf"]
Maybe it.

Yea, I searched if this crash is known. I found a similar issue pop up on the sway project, which went nowhere, since the reporter did not provide any logs. I therefore suspect that this issue may lie in wlroots

nice "I searched" #6398


Custom command switch gets stuck when hibernating windows

I added a custom command to hibernate my machine using the cmd: "shutdown /h".
When I turn the switch ON in Home Assistant the command is executed, hibernating immediatly.
But then the switch does not turn off on its own and toggling the switch does not work. When I subscribe to the state topic I get a retained ON payload.

Thanks for creating an issue!
That doesn't sound right at all. I'll take a look when I can.

I set up hass-workstation-service on 2 computers running identical Win10 versions, vps, antivir etc. Only one of them displays this behaviour, however not consistently enough for any kind of reproducibility.
As an aside, this is where a "TOGGLE" type helper in home assistant would come handy. I have almost 10 "input.boolean" helpers that I need to act as toggles, i.e. they must be always off and only do their thing when I flip them on, at which point they automatically flip off to which end I have to use either 10 automations (or a node-red flow) to monitor their state and flip them to off 0.1 sec after they flip on for any reason.

@SecretiveShell fixed this in #122 so I'll close this and it will be included in next release!


Update ferc-ferc plant matching with ccai implementation.

This PR pulls @katie-lamb's CCAI implementation of the FERC-FERC inter-year plant matching process. This new implementation works well, and seems to be running much faster than the old implementation (~2 seconds vs ~8 seconds on the etl_fast dataset).

This looks good so far! Is it the PCA or the clustering that blows up memory?

@zschira I made some somewhat confusing but hopefully helpful plots to better understand the distance between clusters. With the fittedsklearn Agglomerative Clustering model one can make a scipy dendrogram chart (see this example). To understand what a dendrogram is, I thought this example was helpful. That being said, I'm still not entirely sure how to interpret these plots, and would love some other ideas if I've missed the mark here.
These plots are run on the small subset of data I've been using (2000 records), and shows the clusters that are p=40 merges from the final merge. The y-axis shows the distance at which two nodes are merged into one, indicated by a bracket connecting them. I recommend ignoring the labels on the x-axis, but it basically represents the size of each node. The red horizontal line indicates the threshold I've been using for two clusters to be merged (currently .5). The merges made above this line did it happen and aren't represented in the labels. The merges below this bar are represented by the labels and were clusters that were merged to form a new cluster. I progressively zoomed in on the y-axis so we can see what's going on.
There's a big jump from threshold of 10 to threshold of 2. I think this means that 10 is way too large of a cluster distance threshold.

Zooming in on a y axis of 0 to 1, it still seems like a lot of merging happens at a much smaller distance. Maybe it's an indication that the threshold should actually be lower? Maybe it doesn't matter too much if the threshold is .5 or .1, still thinking about that and not entirely sure what to make of it.

More merging happening in the <.05 range. But that's also to be expected. There's maybe more clustering of larger nodes (bigger on the x-axis), which is good.

It's a little harder to visualize results with the model run on the full dataset, but for the most part it seems like results align with the smaller sample dataset. I ran this with p=20, so 20 merges from the final merge, because it was impossible to tell what was going on with p=40.

@zaneselvans I've spot checked a number of plant id's and so far they've all looked like the same plant to me.
I think doing some focused spot checking using the dendrogram as a guide would be interesting. For example, look at clusters that merge just above our threshold at the 0.5-0.6 range and see if they look like the same plant or not, or do the same just below the threshold, and then maybe zoom way in to clusters with very small distances, and see if we can find any that don't seem to be general matches. I guess generally, it would be interesting to find some cases where we're clearly failing (matching plants that shouldn't be matched or not matching plants that should), and see where/why those might have gone wrong.

@zaneselvans

I'm definitely confused here. With several thousand expected clusters (one for each FERC plant) it seems hard to visualize all of them at once.

You can think of the dendrograms as a sample of the several thousand clusters that are created by the model. In our case, the p=40 parameter is not that meaningful, since I'm instead using a distance threshold to decide when merging should stop.  I think the dendrogram is mostly helpful to understand if the distance threshold for merging is appropriate. As @zschira pointed out, the second step for validating is probably to spot check nodes that are merged right below or right above that .5 mark.
This also verifies why the average distance between records in a cluster is always very small (.05 or less) even when I experiment with a distance threshold in the .2-1 range. The vast majority of merges happen between nodes with a distance <.05, so looking at the average distance between records in a cluster isn't a very helpful metric for verifying whether the threshold is good in the .2-1 range.

Have you made histograms of the cluster sizes in the old vs. new systems?

Yes, currently in the notebook in the CCAI repo. Here's a comparison from the smaller sample dataset (2000 records, I didn't have a screenshot of the full model histogram), where the average new cluster size is ~5.8 and the average old plant_id_ferc1 size is ~4.3. This disparity is more pronounced on the whole dataset, where the new average cluster size is ~7.3.

I'll do some spot checking around the current distance threshold as Zach suggests.

I think the y-axis label might belong on the x-axis?
It's interesting that there's a bump at the very high end of the length spectrum (like, one record for every possible year) in the old version, but not in the new version.  I wonder why that would have happened, and whether we've actually lost some good long time series, or if they were bad for some reason and the new algorithm does a better job of distinguishing them?

I think the y-axis label might belong on the x-axis?

Oh yep, I did that too fast, but switch the label from y-axis to x-axis.

It's interesting that there's a bump at the very high end of the length spectrum (like, one record for every possible year) in the old version, but not in the new version. I wonder why that would have happened, and whether we've actually lost some good long time series, or if they were bad for some reason and the new algorithm does a better job of distinguishing them?

That's a good point, and that disparity at the high end of the length spectrum happens in the full dataset as well. I'll spot check some of those.

I think you might need to add an __init__.py to the new record_linkage subpackge to make it importable, and bring in all of the new modules, and add an import record_linkage into src/pudl/analysis/__init__.py to reflect the new subpackage.

I tried to run the full ETL and ended up with a failure in the plants_steam_ferc1 asset, with the process crashing, so maybe it's running out of memory? I've got 32GB. @zschira how much memory do you have locally? Have you had any recent issues running the full ETL?
Multiprocess executor: child process for step plants_steam_ferc1 was terminated by signal 11 (SIGSEGV).
dagster._core.executor.child_process_executor.ChildProcessCrashException

Stack Trace:
  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.11/site-packages/dagster/_core/executor/multiprocess.py", line 249, in execute
    event_or_none = next(step_iter)
                    ^^^^^^^^^^^^^^^
,  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.11/site-packages/dagster/_core/executor/multiprocess.py", line 360, in execute_step_out_of_process
    for ret in execute_child_process_command(multiproc_ctx, command):
,  File "/Users/zane/miniforge3/envs/pudl-dev/lib/python3.11/site-packages/dagster/_core/executor/child_process_executor.py", line 174, in execute_child_process_command
    raise ChildProcessCrashException(exit_code=process.exitcode)

Giving this a try locally in the context of a full ETL run and:

It's been running for more than 2.5 hours.  The last 40 minutes or so has been only the two assets that are parallelized internally plants_steam_ferc1 and imputed_hourly_demand_ferc714.  Given that each of them runs in 10-20 minutes in isolation, I wonder if they aren't fighting over resources and getting bogged down.
It would be interesting to see what the run times look like if each of these assets is only allowed to use at most 50% of the available CPUs.
For the imputed_hourly_demand_ferc714 asset, there's some internal logging that indicates it's making progress (whenever it starts imputing a new year, it prints out the year).
It would be nice if there were some logging output from the plants_steam_ferc1 asset as well, so we can see what kind of progress it's making, if it's going to take a very long time. Are there any internal steps it can report out about? As it is, it hasn't produced any output for 2.5 hours, and the last thing it said was:
2023-12-01 11:45:34 [    INFO] catalystcoop.pudl.analysis.record_linkage.classify_plants_ferc1:47 Identifying distinct large FERC plants for ID assignment.

Re-running just the plants_steam_ferc1 asset in isolation, I get to the same crash in about 12 minutes.

bad_plants = {
    # APS Yucca 3 misses the plant in 2021-2022 solely due to name change and a blip in fuel fraction splits. Everything else is consistent. Seems too sensitive.
    "yucca3": lambda steam: (steam.plant_id_pudl == 1013),
    # The columbia power station in Wisconsin seems genuinely complicated. It's a mishmash of different units with different owners.
    # Sometimes discontinuous years. Getting multiple records from the same year assigned the same plant_id_ferc1.
    # Sometimes big changes in the capacity within the same plant_id_ferc1.
    "columbia": lambda steam: (steam.plant_id_pudl == 124),
    # APS West Phoenix 5, data for 2021-2022 are not associated with other records due to different name.
    "west_phoenix_5": lambda steam: (steam.plant_name_ferc1.str.contains(r"west phoenix.*5", regex=True, case=False)),
    # sterling avenue plant is very consistent except for construction_type, which is usually null but has values in a couple of years,
    # so in those years it gets split off from all the other years, e.g. in 2003
    "sterling": lambda steam: (steam.plant_name_ferc1.str.contains(r"sterling", regex=True, case=False)),
    # Jeffrey Energy Center is getting split up badly due to minor changes in the name
    "jeffrey": lambda steam: (steam.plant_name_ferc1.str.contains(r"jeffrey", regex=True, case=False)) & (steam.utility_id_ferc1==255),
    "jeffrey_8pct": lambda steam: (steam.plant_name_ferc1.str.contains(r"jeffrey.*8\%", regex=True, case=False)) & (steam.utility_id_ferc1==255),
    # Jeffrey Energy Center NOT getting split up when it should due to large variations in capacity
    "jeffrey_capacity": lambda steam: (steam.plant_name_ferc1.str.contains(r"jeffrey energy cntr", regex=True, case=False)) & (steam.utility_id_ferc1==255),
    # Belews Creek record for 2022 gets lost. The only change is gas/coal fuel ratio
    "belews": lambda steam: (steam.plant_name_ferc1.str.contains(r"belews", regex=True, case=False)),
    # Minor name changes split valmy 1&2 records.
    # NA fuel fractions in 2013 split valmy 1&2 records.
    "valmy12": lambda steam: (steam.plant_name_ferc1.str.contains(r"valmy.*1.*2", regex=True, case=False)),
    # Many cases of more than one record from the same year getting assigned the same plant_id_ferc1
    "valmy_duplicate_years": lambda steam: (steam.plant_name_ferc1.eq(r"valmy")),
    # Minor name change + null fuel fractions split this plant:
    "niles": lambda steam: (steam.plant_name_ferc1.str.contains(r"niles")) & (steam.plant_type.eq("combustion_turbine")),
    # Minor name change and flaky fuel categorization splits the plant
    "manatee": lambda steam: (steam.plant_name_ferc1.str.contains(r"manatee")) & (steam.plant_type.eq("steam")),
    # HB Robinson steam plant records get split due to whitepsace change in name, flaky capacity & fuel fraction reporting.
    "hb_robinson_steam": lambda steam: (steam.plant_name_ferc1.str.contains(r"h.*b.*robinson", case=False, regex=True)) & (steam.plant_type.eq("steam")),
}

@zschira Looking at the "bad plants" that @zaneselvans found, it seems like the lowest hanging fruit are those which have name changes. I wonder if it makes sense to play with the TfidfVectorizer parameters. The old model used character n grams from 2 to 10, but in my experience with FERC to EIA I've had more success with word level tokenization and shorter n grams. Otherwise, some stricter or specific rules in the Company Name Cleaner to address the cases Zane found seems like it would help. It might be worth trying to strip digits from plant names?

Current state - we're tuning parameters to address some of these specific cases, while trying to avoid over-fitting to the specific test cases.
We want to have some sort of metric for when this is "good enough," so we can avoid tuning forever.
Options:

time-box remaining development
flesh out the generative testing that @zschira has already set up - and then use match % as a comparison metric with existing FERC-FERC
potentially use a metamorphic test and its match % to compare with existing FERC-FERC as well - definitely timebox this if we decide to do it at all

It looks like when the modules under pudl.analysis were rearranged, the new import paths were not added to pudl.analysis.__init__.py so they were not importable by other modules, which was causing the use of pudl.analysis.fuel_by_plant functions to fail in the steam table processing.  the record_linkage module was also removed from this subpackage (and turned into its own subpackage) but it wasn't
I'm running the full ETL locally using the etl_full job and also seeing that EPA CEMS is using all 10 of my CPUs rather than limiting itself to a concurrency of 2.  Did something get changed that makes that safe? Is the EPA CEMS concurrency setting not working in the Dagster UI?

After including the new import paths, I get a failure on the plant parts EIA:
AssertionError: Consistency of all matches across years dipped below 75.0% to 53.5%
  File "/Users/zane/code/catalyst/pudl/src/pudl/analysis/eia_ferc1_record_linkage.py", line 94, in out_pudl__yearly_assn_eia_ferc1_plant_parts
    connects_ferc1_eia = prettyify_best_matches(
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/zane/code/catalyst/pudl/src/pudl/analysis/eia_ferc1_record_linkage.py", line 888, in prettyify_best_matches
    check_match_consistency(
  File "/Users/zane/code/catalyst/pudl/src/pudl/analysis/eia_ferc1_record_linkage.py", line 998, in check_match_consistency
    raise AssertionError(

Hmm, attempting to re-run just the EPA CEMS it seems to do them two-by-two. So maybe it was just that the whole EPA CEMS graph job had failed and left the unstarted ghost assets in the UI.

@zschira The builds passed last night so this could go into dev cleanly now.
Do we know why the test coverage drops by half a percent between this PR and dev? That seems like kind of a lot.


feat: add check of minimal recommended zksolc version

What :computer:

Add fetching of recommended minimal zsksolc version as mentioned in #96
Add check for version passed through --use-zksolc parameter that it's greater than minimal
Add an ability to skip this check using ZKSOLC_SKIP_MINIMAL_VERSION_CHECK

Why :hand:

Closes: #96

Evidence :camera:

@dutterbutter my bad, fixed the clippy warnings.
Also I may do the #208 and #42 here too, as they are related

@Velnbur thanks for the contribution! If you intend to pick up those issues please do so in a separate PR.


API should be exposed as proper sort fn

I should be able to do the following: myArray.sort(colorSortFn).

Like CSS Stats does it here: https://github.com/cssstats/css-unit-sort/blob/master/index.js

Fixed in https://github.com/bartveneman/color-sorter/pull/4

Re-opened, because the previous implementation was very buggy.


Database migration test environment

We need an environment where we can test database migrations, e.g. a durnicloud vm. It might look like this:

A growing collection of all database dumps from different environments and version.
A growing collection of Stroom releases.
Core services running in containers.
Flyway CLI, to be used in conjunction with a Stroom release to run the migration. Or we could just run Stroom. We'll probably want to run Stroom anyway, to check that Stroom can make sense of the migrated data and not just the migration scripts work.
Scripts and READMEs to make load dumps and run migrations.
A record of the results of migration tests.

This might not be necessary -- we could just test the environments one-by-one as we upgrade them. When we upgrade environments we'll do it in situ but with backups/dumps, so this might just not be necessary. But checking migration functions is something we can do for all environments immediately, which gives us a head-start on the deployment to these environments.

Also test dumps for all environments!

Collection of scripts in stroom-resources/bin/migration.


Remove legacy-ui folder

Codecov Report

Merging #12 into master will increase coverage by 1.52%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master     #12      +/-   ##
=========================================
+ Coverage    4.39%   5.91%   +1.52%
=========================================
  Files         206     163      -43
  Lines        5961    4426    -1535
  Branches     1204     897     -307
=========================================
  Hits          262     262
+ Misses       4510    3282    -1228
+ Partials     1189     882     -307

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7cc8d0f...c136193. Read the comment docs.


Review My App!

/cc @codepathreview
My team and I have been behind on the app until very recently. We went to a HackNY together and completed most of the required features of the app. Could this be applied to a few of the previous milestones we missed?

:+1: Nice work on the second implementation sprint! There's one week left to complete all of the required stories. Excited to see your progress next week!
Good to see that you have already started adding UI polish to your app. :star2:


Reprosyn Example

Closes #63
Changes made:

ReprosynGenerator class to make it easy to deal with Reprosyn.
Printing stderr on .generate() to help with debugging.
Update to latest reprosyn.
NOTE to simplify interacting with Reprosyn I removed our 'no header' rule in the _parse_cv() and write_to_string() functions in the Dataset class, and also removed the schema parsing in _parse_cv(). This was a quick fix because Reprosyn currently expects a headered-dataset, and does some data wrangling based on the column names (specifically, drops the Person ID column). We will want to change this in the future, but it's fine for an initial workflow.

I merged main into this branch to include some of the changes made to Dataset.
I have however undone the change to Dataset that breaks tests and reverted to the code from main:
data = pd.read_csv(fp, header=None, dtype=dtypes, index_col=None)

I assume that this breaks the notebook and the code from this branch, so either way main will be broken in some way or another. My plan is to re-write parts of Dataset to be able to read datasets with and without header. I am still undecided on whether this change should be done here or as part of a new branch. Will merge or not depending on this.

While playing with the notebook, I've noticed that the output is weird (i.e., all the data is in one column).
There is an issue at the interface between reprosyn and Dataset. I'll probably wait for you to come back to discuss the best way forward.


How often is it necessary to create JAYSON objects?

Open Tests.swift and look at func testBuild().
I understand that root element should be JAYSON but is it always necessary to call JAYSON(JAYSON(JAYSON(...))) instead of inner Swift literals?
For example, could you explain the difference between j["tree1"] = JAYSON(["index" : "myvalue"]) and j["tree1"] = ["index" : "myvalue"]?

Thank you.

I understand that root element should be JAYSON

Yes, that's right.
but, The following objects do not need to be wrapped in JAYSON.
JAYSON
JAYSONWritableType


Fix broken headings in Markdown files

GitHub changed the way Markdown headings are parsed, so this change fixes it.
See bryant1410/readmesfix for more information.
Tackles bryant1410/readmesfix#1

How's it going @bryant1410 Thank you very much for the fix.


Correctly validate the nil check for interfaces.

I found this bug while testing newly generated apigatewayv2 controller where the update code path was not triggered because no delta was being reported for an updated api resource.
The root cause was that string pointers were not being checked correctly for nil value.

To perform nil check on interfaces, only doing i == nil is not sufficient as pointed out in

https://tour.golang.org/methods/12
https://mangatmodi.medium.com/go-check-nil-interface-the-right-way-d142776edef1

I updated the nil check to check for value stored in the interface type as suggested by links above and then confirmed that delta path for apigatewayv2 controller was correctly working.
I also added some unit tests for nil checks on interface types that can have nil values.

We totally missed that Go have typed nils.. Great catch on this @vijtrip2 ! but i still think that we should avoid using reflect as much as possible. Especially in an area that is related to the controllers reconciliation. I believe that we might see decreasing performance in ACK controllers in the future.
IMO it would be better to solve this issue at the "delta code generation" level, by generating nil checking functions for each type, and implementating nil checking functions for built-in types in the runtime/compare package.
Also there is a change that Go 1.17 will bring generics :) which can be a very good solution for this issue.

A small experiment to compare the performance of  == nil with reflect.ValueOf().IsNil():
package main_test

import (
	"reflect"
	"testing"
)

func isNilReflect(i interface{}) bool { return reflect.ValueOf(i).IsNil() }
func isNilEqualOp(s *string) bool     { return s == nil }

func Benchmark_reflection(b *testing.B) {
	var s *string
	v := interface{}(s)
	for i := 0; i < b.N; i++ {
		isNilReflect(v)
	}
}
func Benchmark_equalOp(b *testing.B) {
	var s *string
	for i := 0; i < b.N; i++ {
		isNilEqualOp(s)
	}
}

goos: linux
goarch: amd64
Benchmark_reflection-24         305637628                3.76 ns/op
Benchmark_equalOp-24<PHONE_NUMBER>               0.247 ns/op
PASS
ok      command-line-arguments  1.834s


Fix a few minor issues with example code in docs

enumerate is not actually used
variable name "labels" does not match
prepare method should be called

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

@sgugger Thanks, I applied the change.

Thanks a lot!


update from data_set to dataset argument

for portalcasting v0.32.0

updating app to work with portalcasting v 0.36.0


Improved runtime of has_bad_word

These methods used to have quadratic runtime of O(|Profane Words| * |input_text|) (possible even slower if you consider the runtime of the regex operations within the censor method). Using censor to implement has_bad_word is fundamentally inefficient. I wanted to use ProfanityFilter on my large dataset (millions of YouTube comments) and it was prohibitively slow. My new implementation leverages a dictionary to run in linear time and quits early if it finds a profane word, rather than doing tons of unnecessary computations. The old implementation made little progress in an hour on my dataset, whereas my implementation did the whole dataset in under 2 minutes.

The continuous integration check is failing but I think the problem is not on my end. If you look at the error message, something else is going on.

Hmm, I suppose the weakness of my approach is that it doesn't play as nicely with your self._no_word_boundaries flag. I tried to make it insensitive to punctuation and case, but maybe you have a suggestion on how to make it better. Do you have a suggestion on how to implement Regex here while keeping the linear runtime from the dictionary approach?

Does it actually go much faster? I feel like the GPU model prediction is always going to be the HUGE bottleneck. If it has the same functionality then I'm fine to merge it in, but I'm not as concerned with speed in this case

Is this still being updated to be made compatible?

Hi @aarashy ,
Are you able to add a unit test for what you have suggested.
Regards,
@duttonw


feat: refactor entities

Entities now support a more classical entity pattern, more in line with Dynamo where the partition key and optional sort key must be provided at creation time.
The key is a composite key provided in the following ways:

Tuple - [partition, sort] - Delete/Get
Map - { [partitionField]: value, [sortField]?: value } - Delete/Get
In the value - { [partitionField]: value, [sortField]?: value } - Set

When a key is returned, we always use the Map format (or the in the value format for get).
Caveat 1: the schema is now required as I was struggling to get the types to work with just the type parameters.
Caveat 2: existing tables must be renamed and recreated
const myEntity = entity("myEntity", {
   schema: { key: z.string(), value: z.number() },
   partitionKey: "key" });
const myEntity2 = entity("myEntity2", {
   schema: { part: z.string(), key: z.string(), value: z.number() },
   partitionKey: "part",
   sortKey: "key"});
// to use numbers or binary as keys, provide the type explictly
const myEntity3 = entity("myEntity3", {
   schema: { key: z.number(), value: z.number() },
   partitionKey: { key: "key", type: "number" }});

const { version } = await myEntity.set({ key: "aValue", value: 1 });
await myEntity.set({ key: "aValue", value: 1 }, { expectedVersion: version  });
await myEntity.get({ key: "aValue" });
await myEntity.get(["aValue"]);
await myEntity.delete({ key: "aValue" });

await myEntity.query({ partition: "aValue" }); // without a sort key, only one value is returned.

Show some examples in the PR description?

Not sure what's going on with the way we configure partition/sort keys. Should be an array of field names
const myEntity3 = entity("myEntity3", {
  schema: {
    key: z.number(),
    value: z.number()
  },
  partition: ["key"]
});

Some help with the types for capturing order of PK/SK
type Shape = {
  [attribute in string]: z.ZodType;
};

type Attributes = z.ZodObject<any> | Shape;

type AttributeNames<Attr extends Attributes> = Attr extends z.ZodObject<infer A>
  ? keyof A
  : keyof Attr;

export interface Entity2<
  Attr extends z.ZodObject<any> | Shape,
  Partition extends readonly AttributeNames<Attr>[],
  Sort extends readonly AttributeNames<Attr>[] | undefined = undefined
> {}

export function entity2<
  const Attr extends z.ZodObject<any> | Shape,
  const Partition extends readonly AttributeNames<Attr>[],
  const Sort extends readonly AttributeNames<Attr>[] | undefined = undefined
>(input: {
  attributes: Attr;
  partition: Partition;
  sort?: Sort;
}): Entity2<Attr, Partition, Sort> {
  throw new Error();
}

const a = entity2({
  attributes: z.object({
    key: z.string(),
    id: z.string(),
  }),
  partition: ["key", "id"],
  // sort: ["key", "id"],
});