Extension registration and automation for gardener-operator

How to categorize this issue?

/area usability
/area ops-productivity
/area open-source
/area ipcei
/kind enhancement
What would you like to be added:
The following concept aims at improving the overall experience and productivity for operators. A Gardener installation usually needs additional and tedious preparation tasks to be done, e.g. creating storage buckets for backups or managing DNS entries. All of those can be automated via gardener-operator.
They even overlap with requirements that were already implemented for shoot clusters, but never made it to the Garden due to conceptional reasons.
Therefore, we plan to add a new custom resource Extension:
🎯 Goals

Registration of a provider extension for the Garden-Runtime and Shoot clusters at the same time.

Serves common tasks in Runtime cluster, e.g. reconciling DNSRecord or BackupBucket resources (gardener-operator will need to create those).
Will be translated into and deployed as ControllerRegistration/ControllerDeployment (ref) in Garden cluster.

Registration of an optional admission controller (example).
Stick to Helm deployment logic (ref) but only rely on OCI repositories to fetch charts/assets (similar to Flux's support for OCI Repositories).
Add great defaulting for known extensions, hosted in github.com/gardener. This allows smaller and trial landscapes to be setup with a minimal Extension resources (see Example 2 below).

🙅♂ Non Goals

It's not planned to eliminate ControllerRegistrations. For complex or advanced configurations, operators should still consider their additional usage.

Extension API:
Fields of the Extension API are very similar to ControllerRegistration/ControllerDeployment as requirements and goals overlap for the main part.
Example 1:
apiVersion: operator.gardener.cloud/v1alpha1
kind: Extension
metadata:
  name: aws
spec:
    resources: # optional - defaulted by Operator for well-known extensions
  # - kind: BackupBucket
  #   type: aws
  #   primary: true|false
  #   globallyEnabled: true|false # only valid if kind=Extension
  #   reconcileTimeout: 30s # only valid if kind=Extension
  #   lifecycle: {}
  #   workerlessSupported: true|false # only valid if kind=Extension
  version: # optional - defaulted by Operator for well-known extensions
  deployment: # optional - defaulted by Operator for well-known extensions
    admission:
      ociRepositoryURL: # optional - defaulted by Operator for well-known extensions
      type: helm
      providerConfig:
        values: {}
    extension:
      ociRepositoryURL: # optional - defaulted by Operator for well-known extensions
      type: helm
      providerConfig:
        values: {}
    # policy: OnDemand|Always

The resources configuration merges the extension handling for Garden and Shoot clusters, whereas the most part is irrelevant for the Garden and only needed to craft the ControllerRegistration.
With decent defaulting, we want the extension registration for operators to be as simple as the following:
Example 2:
apiVersion: operator.gardener.cloud
kind: Extension
metadata:
  name: aws

Tasks:

Prerequisites

[x] https://github.com/gardener/gardener/pull/9924
[ ] Introduce class in extensions.gardener.cloud/v1alpha1.DefaultSpec
[ ] Default values for well known extensions, e.g. AWS, GCP, etc. Those should be maintained by the Renovate bot (see hackathon code)

Registration and Deployment

[x] Extension registration in Virtual Garden Cluster for Seeds: https://github.com/gardener/gardener/pull/10006
[ ] Extension Deployment in Runtime Garden Cluster

[x] #10255
[x] #10254
[ ] Enhance extension controller

[ ] Admission controller deployment (example): #10277
[ ] Care controller for periodic health checks
[ ] OwnerRef change for webhook configuration
[ ] Deletion and Required condition logic

Garden controller adaptations

[ ] https://github.com/gardener/gardener/pull/9940
[ ] Do we need extensions.gardener.cloud/v1alpha1.BackupEntry deployment?

Adapt Backup{Bucket,Entry} and DNSRecord extensions (for now, others maybe later) to be able to run in a cluster twice (one resp. for garden, one resp. for seed)

[ ] gardener/gardener-extension-provider-alicloud:
[ ] gardener/gardener-extension-provider-aws:
[ ] gardener/gardener-extension-provider-azure:
[ ] gardener/gardener-extension-provider-gcp:
[ ] gardener/gardener-extension-provider-openstack:

Push Helm charts to OCI registries (for both extension and admission components)

[x] gardener/gardener: https://github.com/gardener/gardener/commit/cb947e812522dd42595012382db0465eeb6b3986, https://github.com/gardener/gardener/commit/75d338ab764e64fa36db607d166367437a90c606
[ ] gardener/gardener-extension-provider-alicloud: https://github.com/gardener/gardener-extension-provider-alicloud/pull/734 https://github.com/gardener/gardener-extension-provider-alicloud/pull/737
[x] gardener/gardener-extension-provider-aws: https://github.com/gardener/gardener-extension-provider-aws/pull/1012 https://github.com/gardener/gardener-extension-provider-aws/pull/1020
[x] gardener/gardener-extension-provider-azure: https://github.com/gardener/gardener-extension-provider-azure/pull/920 https://github.com/gardener/gardener-extension-provider-azure/pull/933
[x] gardener/gardener-extension-provider-gcp: https://github.com/gardener/gardener-extension-provider-gcp/pull/805 https://github.com/gardener/gardener-extension-provider-gcp/pull/815
[x] gardener/gardener-extension-provider-openstack: https://github.com/gardener/gardener-extension-provider-openstack/pull/811 https://github.com/gardener/gardener-extension-provider-openstack/pull/828
[x] gardener/gardener-extension-provider-equinix-metal: https://github.com/gardener/gardener-extension-provider-equinix-metal/pull/317
[x] gardener/gardener-extension-networking-calico: https://github.com/gardener/gardener-extension-networking-calico/pull/445 https://github.com/gardener/gardener-extension-networking-calico/pull/453
[x] gardener/gardener-extension-networking-cilium: https://github.com/gardener/gardener-extension-networking-cilium/pull/369 https://github.com/gardener/gardener-extension-networking-cilium/pull/376 https://github.com/gardener/gardener-extension-networking-cilium/pull/377
[x] gardener/gardener-extension-os-gardenlinux: https://github.com/gardener/gardener-extension-os-gardenlinux/pull/188
[x] gardener/gardener-extension-os-suse-chost: https://github.com/gardener/gardener-extension-os-suse-chost/pull/162
[x] gardener/gardener-extension-os-coreos: https://github.com/gardener/gardener-extension-os-coreos/pull/121
[x] gardener/gardener-extension-os-ubuntu: https://github.com/gardener/gardener-extension-os-ubuntu/pull/143
[ ] gardener/gardener-extension-runtime-gvisor: https://github.com/gardener/gardener-extension-runtime-gvisor/pull/138
[ ] gardener/gardener-extension-registry-cache: https://github.com/gardener/gardener-extension-registry-cache/pull/229
[ ] gardener/gardener-extension-shoot-dns-service: https://github.com/gardener/gardener-extension-shoot-dns-service/pull/359 https://github.com/gardener/gardener-extension-shoot-dns-service/pull/362
[x] gardener/gardener-extension-shoot-cert-service: https://github.com/gardener/gardener-extension-shoot-cert-service/pull/282
[x] gardener/gardener-extension-shoot-oidc-service: https://github.com/gardener/gardener-extension-shoot-oidc-service/pull/222
[x] gardener/gardener-extension-shoot-lakom-service: https://github.com/gardener/gardener-extension-shoot-lakom-service/pull/98
[x] gardener/gardener-extension-shoot-falco-service: https://github.com/gardener/gardener-extension-shoot-falco-service/pull/48
[x] gardener/gardener-extension-shoot-networking-problemdetector: https://github.com/gardener/gardener-extension-shoot-networking-problemdetector/pull/166
[x] gardener/gardener-extension-shoot-networking-filter: https://github.com/gardener/gardener-extension-shoot-networking-filter/pull/163
[ ] gardener/gardener-extension-shoot-rsyslog-relp: https://github.com/gardener/gardener-extension-shoot-rsyslog-relp/pull/147
[ ] stackitcloud/gardener-extension-acl:

/cc @MartinWeindel @rfranzke @oliver-goetz @dkistner

/assign @MartinWeindel @oliver-goetz @timuthy


Replace tuple of scores with ScorePair

Codecov Report
Patch coverage: 70.58% and project coverage change: -0.09 :warning:

Comparison is base (a3900cd) 96.92% compared to head (434c653) 96.83%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #161      +/-   ##
==========================================
- Coverage   96.92%   96.83%   -0.09%
==========================================
  Files          72       73       +1
  Lines       11645    11663      +18
==========================================
+ Hits        11287    11294       +7
- Misses        358      369      +11

Impacted Files
Coverage Δ

eval/src/lib.rs
100.00% <ø> (ø)

eval/src/score_pair.rs
60.00% <60.00%> (ø)

eval/src/complex.rs
96.75% <100.00%> (-0.12%)
:arrow_down:

... and 1 file with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Create TAROT Handbook chapters and intro

Almost done. Check it out here

[ ] Photo and video of shuffle techniques
[ ] tasty visuals

editing in changes from Vanessa


env: support runtime evaluation

An attempt at getting rid of the call to builtins.getEnv to determine the current working directory. Taking inspiration from the implementation in https://github.com/numtide/devshell.
Things to implement:

[ ] Replace all remaining eval-time uses of environment variables (${config.env....})

Things to invent:

[ ] Find a way to splice environment variables into config files that don't support environment variable substitution
[ ] Find a way to make this work for containers
[ ] Find a way to deal with ordering (for example, AN_EXAMPLE_DATA_DIR gets set before DEVENV_STATE, thus cannot reference it with $DEVENV_STATE/example)

Nice! I'd love to see it working :)

What I'm worried the most as a blocker here is putting real paths into configuration files generated by Nix.
That's really important and I don't see how it can be done, since you need to resolve the env var as bash to be able to generate the config file.
Maybe we need to use recursive Nix or IFD?

It might be necessary to implement a custom templater, similar to what NixPak does with Sloth values and the NixPak launcher. Sloth values are a mini-DSL for defining "thunks" of strings that get rendered at runtime.
For example, the Nix expression (or a string that gets parsed with Nix code and converted to this, think envsubst)
sloth.concat [
  "unix:path"
  (sloth.env "XDG_RUNTMIE_DIR")
  "/nixpak-bus"
]

gets turned into an intermediate JSON representation (hypothetical, the NixPak implementation outputs something slightly less efficient since I'm bad at Go :P)
{
  "items": [
    "unix:path="
    {
      "key": "XDG_RUNTIME_DIR",
      "type": "env"
    }
    "/nixpak-bus"
  ],
  "type": "concat"
}

which the launcher turns into a string at runtime, inspecting its environment to discover that the value of the environment variable XDG_RUNTIME_DIR is /run/user/1000:
unix:path=/run/user/1000/nixpak-bus

I imagine we could have Nix generate config files in the intermediate representation and then have a wrapper binary render them into a file in $DEVENV_STATE at runtime, inserting the path of the rendered file, either into another file or into the command-line arguments of a process.

I don't think this is going to land, because we have made the decision that env vars should be expanded at the Nix language.


Upgrade repoID to int64, because that's the size from GitHub

Summary
Changes the stored and protocol repoID to be int64 size instead of int32 size.  go-github uses an int64 for this field, and GitHub doesn't guarantee the int size, which appears to be sequential and was 734857594 for a recently-created repo.
Change Type
Mark the type of change your PR introduces:

[x] Bug fix (resolves an issue without affecting existing features)

Testing
Ran go test ./... and ensured no compilation errors.  This should include an up-migration, but not a down-migration (which seems scary to me).
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[x] Added comments to complex or tricky code sections.
[x] Checked that related changes are merged.

coverage: 37.469% (+0.004%) from 37.465%
when pulling ac3de70a4b47b952157f50b3028b30ec684a4dcd on evankanderson:supersize-my-repo-id
into f846a56be214fee19d908ca6b6eb68b48ea7700d on stacklok:main.


Add OpenMapTiles demotiles for more complex testing

This is an attempt make more complex demo tiles for https://github.com/maplibre/maplibre-gl-js/issues/198
I have created omt tiles of the N047E011 region using planetiler, extracted them with mbutil, uncompressed the pbf files, and put them into the 'tiles-omt' directory. I have also added osm bright based styles and added the fonts this style requires. I added in spites which I created for my own project which are a based on open sources and a little more complex than the ones that come with osm bright.
I have created a test here
https://github.com/WifiDB/demotiles/tree/gh-pages/
I have created two demo pages here
https://wifidb.github.io/demotiles/tiles-omt/ (new omt demo tile with a osm-bright-gl style)
https://wifidb.github.io/demotiles/terrain-tiles/ (same with terrain and hillshade added)
Here are some screenshots in use

Are all the things added here "copyright" free?

I tried to keep or add license information where it was required.
The tiles I generated myself and the contain the proper OpenMapTiles/OpenStreetmap attributions.
The sprites I generated myself with https://github.com/acalcutt/icon-fonts . As far as I can tell the licenses for the svgs used are open, you can see where they all came from and their licenses here ( https://github.com/acalcutt/icon-fonts/tree/master/svg/furkot )
The fonts came from https://github.com/openmaptiles/fonts
The style is https://github.com/openmaptiles/osm-bright-gl-style with some modifications.

Looks like the Noto fonts are OFC License
The style is BSD 3-Clause License with CC-BY 4.0. The attribution for OpenMapTiles is included in the tiles
I guess the sprites could be simplified to the ones that come with osm-bright-gl-style, since that could make licensing easier.

I changed the sprites to use the ones included with osm-bright-gl-style and moved them to inside the style folders so they could be covered by the same openmaptiles LICENSE.md file.  These are a lot more basic, but at least they have a more clear license.

Also, I know this is hard to review because of the amount of files, but that can't really be helped because of the tiles and fonts added.
If you look at https://github.com/wifidb/demotiles/tree/ml_tiles
What I have added is the following
** Fonts used by the osm-bright style **
font/Noto Sans Bold
font/Noto Sans Italic
font/Noto Sans Regular
** Styles based on osm bright **
styles/osm-bright-gl-style
styles/osm-bright-gl-terrain
** OMT vector tiles pbf files **
demotiles-wdb\tiles-omt(0-14)
** Tilejson Generated by tileserver-gl **
tiles-omt\tiles.json
** Demo Pages **
tiles-omt\index.html
terrain-tiles\index.html
I thought about moving the root directory style.json into the styles folder, but I didn't want to change what was already in place. might be something to do in a future PR though.
Let me know if there are any other concerns or recommendations.

Besides licenses and clone time, I can't think of other things...

I think the license should be fine now, no?
I offer the tiles i generated without a license beyond the attibutions required.
the font and styles both have a permsissive open license that i included inside the directories. the font OFL license allows modification and redistibution. the style allows redistibution with attribution, which I have here.
I also think the checkout will end up faster than it is on my fork becusse i have a few branches with a lot more tiles that this wont have (xheckout on my fork checks out 50,000 files, bit this only adds about 5,000)

Ok, let's merge, we can always revert.

Nice, it looks like it is working at these urls :-)
https://demotiles.maplibre.org/tiles-omt/
https://demotiles.maplibre.org/terrain-tiles/

looks like it took me about 30 seconds to checkout the repo, so it doesn't seam much slower than before...awesome.


Which node.js-version is required?

Ubuntu 12.04 distributes node.js version 0.6 which cannot run node_redis. Maybe add an "engines" field to package.json that specifies the lowest required node.js-version?

In order to better support this project and its new group of collaborators under a new org we are trying to clean up and close issues older than 6 months. Your issue may have been fixed since the issue was open, however, if you are still experiencing a problem please re-open this issue and we will label it accordingly.

The engine will be added in the next version.


grt creates guides that go through blockages

Describe the bug
The attached test case shows guides that go through a blockage. M7 is completely blocked, but some guides go right through it from M5 through to M8:

Expected Behavior
Guides should not cross through obstructions
Environment
Git commit: edcd14ffec5d232d98e8b0c34315b1ce233b1af6
os: Fedora Linux 37 (Workstation Edition)
cmake version 3.26.0-rc5

To Reproduce
Test case:  grt-through-blockage.tar.gz
Relevant log output
No response
Screenshots
No response
Additional Context
No response

@antonblanchard would you give some more context as to what you are seeking to have happen by blocking m7?

@antonblanchard would you give some more context as to what you are seeking to have happen by blocking m7?

Trying to build simple test cases to exercise specific parts of drt. In this particular case I could obstruct all layers other than the two I'm interested in.

What does it mean to block all but two non-adjacent layers?

What does it mean to block all but two non-adjacent layers?

To force specific vias to be used in test cases. I'm not sure how much of an issue this is outside of test cases, and I can see how grt would struggle since it starts with a 2D view of the capacity.


Encoding issues on mips, powerpc, s390x

Hi there,
while building the Debian packages for 1.9.3, the tests found some encoding errors on the mips, powerpc and s390x architectures.
On mips & powerpc:
  1) Failure:
SQLite3::TestEncoding#test_select_encoding_on_utf_16 [/«PKGBUILDDIR»/test/test_encoding.rb:23]:
Expected: 1
  Actual: 0

On s390x:
  1) Failure:
SQLite3::TestEncoding#test_insert_encoding [/«PKGBUILDDIR»/test/test_encoding.rb:42]:
Expected: "foo"
  Actual: "\u6600\u6F00\u6F00"

I hoped you could shed some light on these test failures.
Detailed build logs can be found at:
https://buildd.debian.org/status/fetch.php?pkg=ruby-sqlite3&arch=mips&ver=1.3.9-1&stamp=1396866955
https://buildd.debian.org/status/fetch.php?pkg=ruby-sqlite3&arch=powerpc&ver=1.3.9-1&stamp=1396865677
https://buildd.debian.org/status/fetch.php?pkg=ruby-sqlite3&arch=s390x&ver=1.3.9-1&stamp=1396865788
(All of this happens under ruby2.0.)

I've applied this patch on master.  Thanks!


In hover type information, how can I avoid this?

How can I avoid such monstrosity without adding redundant imports? I'd like the IDE to show me  only the last part of the typenames, especially in this case, where those types were actually  imported from within another "internal" file in my own project.

Does importing the module cause the type to be shortened?

@bubba Here is a better example:
module Main where

import Data.Map.Strict (fromList)

main :: IO ()
main = do
    let a = fromList [("", "")]
    putStrLn "hello"

Hover over fromList and you will see the monstrosity fromList :: [([Char], [Char])] -> Data.Map.Internal.Map [Char] [Char].
After adding Map to  the import list, hovering results in the more saner fromList :: [([Char], [Char])] -> Map [Char] [Char]. The problem is that now (with -Wall) it complains about  a redundant import of Map.
Is it possible to get cleaner hover info without introducing redundant imports?

I presume GHC or whatever is fully qualifying the names of types since they are not imported, to avoid ambiguity. This is probably the right thing, but we could have a settings toggle to control this. Thoughts? @alanz @fendor


Fix parse_beast_trees

From @richelbilderbeek on August 10, 2016 12:35
Currently, parse_beast_trees does not respond correctly when it is asked to parse a non-.trees file.
Copied from original issue: richelbilderbeek/Cer2016#118

The problem is in ape::read.nexus that does not terminate elegantly when given a non-nexus file. I'm working on fixing this upstream, in the ape repository.

At the moment, ape has a workflow that is too old-school for me, using manual tests. I have volunteered to add Travis CI, but this did not get accepted.


Feature/cls compliant

I added the [assembly: CLSCompliant(true)], since the assembly is CLS compliant this make sense.

Please rebase this PR onto the latest upstream/master instead of a merge commit. A PR should only contain commits that contribute something so it's clear what it changes.
Also strip out the bump of version in AssemblyVersion("3.25.0")]. In latest master this has been set back to 1.0.0 as it should only be bumped as part of building and publishing nugets.

I've tried rebase-ing (git rebase upstream/master). It's the first time I've done a rebase so I might have done something wrong but the end result made no changes to the code (which was expected since the merge essentially did the same thing in terms of code changes). When I merged I changed the version number to 1.0.0 so the actual code changes should be correct in this pull request.
Will rebase instead of merging the next time.

Alright, you should do an interactive rebase so you can remove the merge commit from the rebased commits (only rebasing the CLSCompliant commit on top of master). No worries though.

Thanks!


"Strip tracking parameters" works too slow in Safari browser

It works but it takes 1-2 minutes to load the site.

This:
http://example.org/?utm_source=123

core/pull-requests/1097


Fix ps_facetedsearch - bad display after clearing a filter of no result

When using module ps_facetedsearch, if we apply a filter that returns no product, then apply another filter that returns some products, the product box alignment will be broken. All other filters with not empty result will be affected till we refresh the whole page.

Questions
Answers

Description?
In this PR Files Changed: line 194 (196) makes $(prestashop.themeSelectors.listing.product) returns Undefined when we have a no product result (list with no product). Then line 191 will set Undefined class for all 'product' in the next filter and break the alignment.  My solution is separate renderedProducts and productSelectors, if productSelectors is empty by whatever reason, we can use renderedProducts instead. const productSelectors is used locally in function updateProductListDOM(data) so its modification won't affect any other functions.

Type?
bug fix

BC breaks?
no

Deprecations?
no

Fixed ticket?
PrestaShop/PrestaShop/issues/28966

How to test?
After PR, make sure you rebuild theme asset, clear web browser cache, then follow the above issue's steps.

Possible impacts?
Please indicate what parts of the software we need to check to make sure everything is alright.

Maybe it is safer to save productSelectors.first().attr('class') in a variable before removeClass() at line 191.

Maybe it is safer to save productSelectors.first().attr('class') in a variable before removeClass() at line 191.

It fails silently so, it should be fine

@sLorenzini I even can not install a fresh PS1.7.8.x since <IP_ADDRESS> release (I must use git checkout <IP_ADDRESS> to make my development environment in fact :joy:). I can not imagine how a change in a JS file can break other tpl.php file either.
So if you confirm the error only happen after add the PR to a 1.7.8.x refresh install, then I can not check any more :pray:

@sLorenzini I even can not install a fresh PS1.7.8.x since <IP_ADDRESS> release (I must use git checkout <IP_ADDRESS> to make my development environment in fact 😂). I can not imagine how a change in a JS file can break other tpl.php file either. So if you confirm the error only happen after add the PR to a 1.7.8.x refresh install, then I can not check any more 🙏

hello @leemyongpakvn,
Thank you for your feedback.
@PrestaShop/prestashop-core-developers, @florine2623 and I reproduce the same error on 1.7.8.x with a fresh install. Could you help and/or check this error please 🙏?
Thanks

The issue seems to be fixed for the 8.0.x. I think we can validate this PR @sLorenzini but If we want the fix to be included in the next 1.7.8 patch the fix need to be done in the core files. @leemyongpakvn Do you think you could make another for the core? A patch will be released soon, I would be nice to include your work :)

@leemyongpakvn
as this theme min. compat has been set to PrestaShop 8.0.0 you'd need to make changes here:
https://github.com/PrestaShop/PrestaShop/tree/1.7.8.x/themes/classic

@MatShir , @kpodemski Done, my first PR in Core repo :)

Perfect thanks !

Hello,
Thanks @MatShir  for your answer.
Thanks @leemyongpakvn for your PR for 1.7.8 version 🚀
It's ok for the QA then.
Retested for 8.0.0 version and all is good.
QA approved ✅
Thank you

Thank you everyone :)


Add LxMLS 2023

Source: http://lxmls.it.pt/2023/#

LxMLS exists under the recurring summer schools


About selecting demonstration when testing IKE

In function apply_ike_to_multimodal_model in EasyEdit-main/easyeditor/models/ike/ike_main.py, the last line is
icl_examples.append(f'New Fact: {new_fact}\nPrompt: {new_fact}\n\n')

I'm a little confused. Why adding the question and answer of a testing sample when applying ike? My understanding is that the input of model should not contain the true answer of the testing question.

Thanks for your attention to our work.
It seems that you have mentioned this question before in #215, we just follow the setting of IKE. I'd suggest direct contact with the authors to seek further clarification. They may be better poised to assist you in your inquiry.

Thank you for your help. I'll do experiments to explore the influence.


Add build whilst tightly coupled with kubeadm code

glide update failes, glide install will work, then checkout ./vendor (to get patched file).
Either sort out dependancies or get PR's in to kubeadm to support all phase commands and broken bits...

Fixed by #20


removed gradient and added blur

somehow backdrop blur didn't apply to the sidemenu in this branch

I think the mobile menu bg can be fixed by remove the h-full class

this would work but it would have a stopping point for the navbar, not sure how i feel about this.
thoughts? @juancwu @norachams @teoh4770

it's weird XD


October PR

Change Log

Added dockerfile to build dev environment
Added optional proxy variables in dockerfile
Removed unused assemblies
Added architecture diagram

Issues

Issue #1

Approved
Closes Issue #1


Weight conversion help

Trying to figure out how to use your weight conversion script.  This is what I'm trying:
(1) Using GPTQ-for-LLaMa repo to convert LLaMA 65B model to .safetensors file:
$ python llama.py ~/models/llama-hf c4 --wbits 4 --true-sequential --act-order --groupsize -1 --save_safetensors ~/models/llama-hf/llama65b-4bit-nog.safetensors
(2) Using GPTQ-triton repo to convert .safetensors file to compatible version:
$ python convert_weights.py --model ~/models/llama-hf --quant ~/models/llama-hf/llama65b-4bit-nog.safetensors --output ~/models/gptqt_model
Getting this error:
/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/safetensors/torch.py:99: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  with safe_open(filename, framework="pt", device=device) as f:
/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/torch/storage.py:899: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  storage = cls(wrap_storage=untyped_storage)
Traceback (most recent call last):
  File "/home/catid/sources/GPTQ-triton/convert_weights.py", line 87, in <module>
    main()
  File "/home/catid/sources/GPTQ-triton/convert_weights.py", line 58, in main
    if state_dict[name + '.bias'] is not None:
KeyError: 'model.layers.0.self_attn.q_proj.bias'

GPTQ-triton is not currently compatible with transformers HEAD.  You'll need to use a transformers commit before 7dcd870.  For example, pip install git+https://github.com/huggingface/transformers.git@5506d0496957cde19318eee3d34ee682b654abe8.
I've been holding off on compatibility because there's a performance regression in transformers after commit 7dcd870.
Sorry for the rough edge here.

Looks like you mean that I should run the llama.py GPTQ script again after changing to this version of transformers, since that doesn't fix the convert_weights.py script.  It's crunching...

Right, yes.  Let me know if that fixes things.

This seems to emit a different error:
(gptq) ➜  GPTQ-for-LLaMa git:(triton) ✗ python llama.py ~/models/llama-hf c4 --wbits 4 --true-sequential --act-order --groupsize -1 --save_safetensors llama65b-4bit-nog-old-transformers.safetensors
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████| 14/14 [01:53<00:00,  8.12s/it]
Found cached dataset json (/home/catid/.cache/huggingface/datasets/allenai___json/allenai--c4-6fbe877195f42de5/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
Found cached dataset json (/home/catid/.cache/huggingface/datasets/allenai___json/allenai--c4-efc3d4f4606f44bd/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)
Starting ...
Traceback (most recent call last):
  File "/home/catid/sources/text-generation-webui/repositories/GPTQ-for-LLaMa/llama.py", line 457, in <module>
    quantizers = llama_sequential(model, dataloader, DEV)
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/catid/sources/text-generation-webui/repositories/GPTQ-for-LLaMa/llama.py", line 54, in llama_sequential
    model(batch[0].to(dev))
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 765, in forward
    outputs = self.model(
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 614, in forward
    layer_outputs = decoder_layer(
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/catid/sources/text-generation-webui/repositories/GPTQ-for-LLaMa/llama.py", line 49, in forward
    cache['position_ids'] = kwargs['position_ids']
KeyError: 'position_ids'

Should I be on a specific branch of GPTQ-for-LLaMa?  Currently using the triton branch.  Maybe this script should be in your repo so you can version it in a compatible way?

Hmm, I guess GPTQ-for-LLaMa doesn't like the older transformers version. Quite the pickle.  Yeah, I've been meaning to absorb the quantizer script.
I'll get a better fix out, but that'll be later.  I'm right in the middle of something.
Commit 19c0535d792d7e388e7fe799f8cfa350ce74fa9a of GPTQ-for-LLaMa should work; that's what I have on my machine at the moment.
Or you can try to patch convert_weights.py.  Then you shouldn't need to re-quantize the weights.  Off the top of my head this might work:
Replace this entire block:
		if state_dict[name + '.bias'] is not None:
			print(f"Converting bias for {name}")
			state_dict[name + '.bias'] = state_dict[name + '.bias'].to(torch.float16)

with just:
		state_dict[name + '.bias'] = None

Thanks for the suggestion but looks like it needs more surgery:
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/safetensors/torch.py", line 222, in _flatten
    raise ValueError(f"Key `{k}` is invalid, expected torch.Tensor but received {type(v)}")
ValueError: Key `model.layers.0.self_attn.q_proj.bias` is invalid, expected torch.Tensor but received <class 'NoneType'>

Going to try running GPTQ again with the older branch of their repo.

This worked, now just trying to figure out how to get it to run on two GPUs...

Nice to hear it's working now.  I haven't tried multi-GPU setups yet.  At the very least you'll need to do load_quant(filepath, device=None, warmup_autotune=False) when loading the model, otherwise it'll try to move the model to the GPU itself (1).  Then you're welcome to divide the model up for multi-GPU however you'd like.  At least text-generation-webui does this:
                max_memory = accelerate.utils.get_balanced_memory(model)

            device_map = accelerate.infer_auto_device_map(model, max_memory=max_memory, no_split_module_classes=["LlamaDecoderLayer"])
            print("Using the following device map for the quantized model:", device_map)
            # https://huggingface.co/docs/accelerate/package_reference/big_modeling#accelerate.dispatch_model
            model = accelerate.dispatch_model(model, device_map=device_map, offload_buffers=True)

(1) FYI: This disables the warmup_autotune during loading, which won't cause problems, but will cause inference speed to initially lag since it's doing autotuning on the fly.

The autotune was an issue so it's nice to be able to disable it and not worry about fixing it.
Using your favorite branch of transformers, I get this error inside fused_attention.py:
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/gptq_triton/fused_attention.py", line 103, in forward
    query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, offset=offset)
  File "/home/catid/mambaforge/envs/gptq/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 132, in apply_rotary_pos_emb
    q_embed = (q * cos) + (rotate_half(q) * sin)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Specifically:
query_states.device = cuda:0
key_states.device = cuda:0
cos.device = cuda:1
sin.device = cuda:1

I patched that part and then it fails inside the transformers library, so upgraded to latest version of transformers and hit TypeError: forward() got an unexpected keyword argument 'position_ids'
Seems like a bridge too far, so going to use the GPTQ-for-LLaMa older triton codebase rather than the bleeding edge for my project.

Thank you for giving the project a try and giving valuable feedback. I'll take a look at the multi-GPU stuff when I get a chance.

FYI:
As of the latest commit (3daf413123cd55c600966492775f717eb3ac01e0) nightly transformers is supported again, so no need to futz around with weird versions.  I've also added quantized.py to this repo, so there's no dependency on GPTQ-for-LLaMa to do the quantization, and no need for the conversion script.  multi-GPU is still in progress.


Regarding use of DTemp module of LBL

Hello,
I want to use the DTemp module for a data set from MAROON-X. As I have understood in the maroonx_starname.py file I have to write this two lines:
rparams['RESPROJ_TABLES'] = dict()
rparams['RESPROJ_TABLES']['DTEMP3000'] = 'temperature_gradient_3000.fits'
But, this "temperature_gradient_3000.fits" file should be stored in the data directory, as you mentioned in your website. Now in my installation directory of lbl "/home/lbl/models" I could not find this fits file. Moreover when from your website, I am trying to download them, those can not be downloaded.
Can any one help me with this problem?

Did you try running your wrap file?
These 'temperature_gradient' files should be downloaded automatically when you next run the wrap file with the lines you quote above uncommented
rparams['RESPROJ_TABLES'] = dict()
rparams['RESPROJ_TABLES']['DTEMP3000'] = 'temperature_gradient_3000.fits'

If you still have problems after that you can download them manually (tar files are provided at the link you give)
To avoid any confusion for others I'll update the documentation:
By default we add the 3500K (no v sin i) gradient spectra to the wrap files (if run using lbl_setup) to enable Dtemp you just need to uncomment the following lines:

      rparams['RESPROJ_TABLES'] = dict()
      rparams['RESPROJ_TABLES']['DTEMP3000'] = 'temperature_gradient_3500.fits'

Once these lines are uncommented all "no v sin i gradient spectra" are downloaded into the {DATA_DIR}/models directory (when you next run the wrap script).

Note if you do not uncomment these lines you will see no models in the {DATA_DIR}models directory.

After running lbl_setup I have to add the two lines manually to the maroonx_starname.py file.
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
One query I have that, while calling the 'temperature_gradient_3500.fits' we should select the temperature such that the Teff of the star is nearest to the grid values that you have [e.g 3000K to 6000K with a step size of 500K]. Right?
By writing this lines:
    rparams['RESPROJ_TABLES'] = dict()       rparams['RESPROJ_TABLES']['DTEMP3000'] = 'temperature_gradient_3500.fits'
You choose the template which is having a temperature value of 3500K. Now there is another parameter in the warp file rparams['OBJECT_TEFF'] = [3300]
should I put the actual Teff of object here or the temperature that I choose for DTemp i.e. 3500K. Also Why you write the "DTEMP3000" in the uncommented lines? If a star having temperature of 5500K, Then I Should choose the 'temperature_gradient_5500.fits' and the argument should be ['DTEMP5500']. Right?
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Also when I have a prior idea about the vsini value of the star, How I should change this code? [I have seen probably you have the 0 - 19 km/s vsini fits]
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
@njcuk9999 @eartigau

Here is a draft of the explanation I intend to put on the LBL website for future users needing the info. Please tell us if it clear enough. I'll add info on the vsini later.
General considerations regarding temperatures in the LBL and DTEMP
Stellar temperatures are relevant inputs to optimize the LBL outputs and these need some context to
avoid overinterpretation of the requirements. Lets say one attempts to analyze an M dwarf,
for example GJ436. The simbad temperatures vary quite a bit, with values ranging from 3200 to 3500,
with even higher values (probably unrealistic). Lets say that your favorite temperature is 3250 K.
The first place where temperature is relevant is the rparams for your LBL wrap. You can input a value
as accurate as you want, so you may as well input :
rparams['OBJECT_TEFF'] = [3250]
In practice, this is used at the time of constructing the mask and determining the systemic velocity
of the system. The codes rounds this value to the nearest 100 K (here it would be 3300 K) and cross-correlates
the mask with a Goettingen model at that temperature (available per 100 K bins, hence the rounding). This
cross-correlation of a mask of hitherto unknown systemic velocity with a model of (assumed) zero velocity
allows us to find the systemic velocity and this is used when producing the RDB tables at the end. If
you entered a temperature that is off (say 4000 K), this would affect mildly the systemic velocity (wrong
convective blueshift for example), but would have not other impact. Overall, put the best estimate of the
stellar temperature and even an estimate from spectral type is fine.
The other temperature input is the one regarding the DTEMP. As explained in Artigau et al., the gradient of
temperatures are determined as a function of temperature. There's nothing wrong with having more than one
DTEMP temperature gradient. We are still trying to understand the physical meaning of the DTEMP values computed
at temperatures that differ from the bulk temperature of the star. Conceptually, on a mid-M dwarf, one could
look for spectral features of a brighter (say 5000 K) star appearing on the spectral and may want to compute the
DTEMP at 5000K on a cooler object. Note that this is still work-in-progress on our side. So overall, for a T=3250K
object, you could have the DTEMP computed at 3000 K and 3500 K and, out of curiosity, you could also compute the
DTEMP at 5000K. Your rparams would therefore include :
rparams['RESPROJ_TABLES'] = dict()
rparams['RESPROJ_TABLES']['DTEMP3000'] = 'temperature_gradient_3000.fits'
rparams['RESPROJ_TABLES']['DTEMP3500'] = 'temperature_gradient_3500.fits'
rparams['RESPROJ_TABLES']['DTEMP5000'] = 'temperature_gradient_5000.fits'

Thank you for your detailed explanation. I have understood your point.
Still after adding the required lines in warp file I am having some errors while running DTemp.

2024-12-02 11:09:31.180|G| We construct the spline for the "DTEMP3000" response
2024-12-02 11:09:31.181|E| RESPROJ Table DTEMP3000 does not exist
Traceback (most recent call last):
....
lbl.core.base_classes.LblException: RESPROJ Table DTEMP3000 does not exist

Should I copy the repository of LBL again from GitHub?

Can you link your wrap file and the command you are running to run LBL.

Here is my warp file link: https://drive.google.com/file/d/1Y1nXUzMv6YHlySegfYoV_GeOxE1nBLY-/view?usp=sharing

So with version 0.64.015 (and adding some new parameters that are required for this version) your wrap file works for me.
You can check by checking the lbl/lbl/version.txt for the version.
I would recommend:

updating to lbl 0.64.015 (git pull followed by pip install -U -e .  from inside the lbl git directory should be sufficient) but you can delete the directory and start again following the full instruction on the website if you prefer.
Regenerating the wrap file (using lbl_setup)
Re-running the wrap file

If you still get an error after doing this could you please post the full set of messages you get after running the wrap file.

Note that I recommend a new wrap file for new versions as we add new parameters that are required for operation. You can then compare it to then one you sent to make your new wrap file, from the look of if when you generate a new wrap file all you'll have to change is the following:
    # The data directory where all data is stored under - this should be an
    #    absolute path
    rparams['DATA_DIR'] = '/home/arjun/Downloads/GJ436/lbl_test_run_red'

   ...

    # The data type (either SCIENCE or FP or LFC)
    rparams['DATA_TYPES'] = ['SCIENCE']
    # The object name (this is the directory name under the /science/
    #    sub-directory and thus does not have to be the name in the header
    rparams['OBJECT_SCIENCE'] = ['GJ436']
    # This is the template that will be used or created (depending on what is
    #   run)
    rparams['OBJECT_TEMPLATE'] = ['GJ436']
    # This is the object temperature in K - used for getting a stellar model
    #   for the masks it only has to be good to a few 100 K
    rparams['OBJECT_TEFF'] = [3300]

   ...

    # Rotational velocity parameters, should be a list of two values, one being
    #     the epsilon and the other one being the vsini in km/s as defined in the
    #     PyAstronomy.pyasl.rotBroad function
    rparams['RESPROJ_TABLES'] = dict()
    rparams['RESPROJ_TABLES']['DTEMP3000'] = 'temperature_gradient_3000.fits'

Hello Neil, Thank you. It is working now.
It is now working. How can  get to know about the outputs  and their formats from DTemp so that we can further analyse the output.
.

Okay Thank you.
Regards,
Arjun
On Wed, Dec 4, 2024, 12:53 AM Neil Cook @.***> wrote:

We haven't had time to write any more than is in the papers (LBL
https://ui.adsabs.harvard.edu/abs/2022AJ....164...84A/abstract and DTemp
https://ui.adsabs.harvard.edu/abs/2024AJ....168..252A/abstract) and
what is currently the website currently i.e. here
https://lbl.exoplanets.ca/output-data.
—
Reply to this email directly, view it on GitHub
https://github.com/njcuk9999/lbl/issues/60#issuecomment-2515398010, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/BHMBOFGQYK4NB7JUK735NST2DYAMXAVCNFSM6AAAAABSWK7YPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJVGM4TQMBRGA
.
You are receiving this because you authored the thread.Message ID:
@.***>


Rosetta: investigate negative balance error

Error: unable to process blocks: unable to process block: negative balance -132291985022:&{Symbol:FLOW Decimals:8
 Metadata:map[]} for &{Address:8624b52f9ddcd04a SubAccount:<nil> Metadata:map[]} at &{Index:14896644
 Hash:c1135fa9c3df176b3e84e7c8f2b527c14150a03c60e81750c158f9b7ec737eca}: unable to add block to storage
 c1135fa9c3df176b3e84e7c8f2b527c14150a03c60e81750c158f9b7ec737eca:14896644: unable to sync to 15791893

Looks like a potential overflow issue to me.

Rosetta CLI fetches the balance for accounts it sees for the first time. After that, it does accounting internally and adds/subtracts amounts for operations it sees along the way. Eventually, it observed that the operations involving this specific account resulted in a negative balance.
However, this is a special case as the address 0x8624b52f9ddcd04a is the address of the FlowIDTableStaking contract. When checking the balance for that address, value of<PHONE_NUMBER>676 is always returned, so it is not affected by the operations as a normal account would be.
Simple way to handle this is to add this address to a list of exempt accounts, for which Rosetta CLI will not investigate balance changes or do reconcilliation.
Content of exempt-accounts.json:
[
    {
        "account_identifier": {
            "address": "8624b52f9ddcd04a"
        },
        "currency": {
            "symbol": "FLOW",
            "decimals": 8
        }
    }
]

Differences of my rosetta configuration file, compared to the default rosetta configuration:
exempt_accounts - path to the exempt-accounts.json, relative to the main configuration file
reconciliation_disabled - true
ignore_reconciliation_error - true (maybe unnecessary considering the config above)


Authorising capability for PREFETCH.I in integer mode

The spec says that the authorising capability for a PREFETCH.I is:

The cs1 register in capability pointer mode
ddc in integer pointer mode

However, ddc is a data capability, so it would make more sense to have pcc (that always has X-permission) authorising the PREFETCH.I in integer pointer mode.

@arichardson @jrtc27 : Do you have a view on this?

PCC sounds right to me.

I also think PCC makes more sense here.

Ok, thanks for your comments! I will make a PR to address this.


Context Sensitive Emoji Menu

It would be super nice if I type a sentence "I will see you at home" and press WIN+. to open the emoji menu then one of the top results is already if I would have searched for the last word that I typed.
Bonus points if I could go back anywhere in the sentence and do the same thing replacing things with emojis. Obviously, my most used emojis should still be near the top as well (different smiley's).
If you are wondering why someone would care so much about emoji? I live in Sweden and speak Swedish fluently but my native language is English. The Swedish language compared to the English language is quite poor in providing synonyms for words to account for different connotations and subtle nuances. So Swedes tend to use emoji's a lot to convey those emotions and nuances instead.
So even my business emails contain emojis 😁

Hi,
Thank you for the suggestion.
I'm adding the "Suggested PowerToy" label, as it may be possible to achieve it through a PowerToy.
In any case, the place to send generaç Windows suggestions would be through the Feedback Hub: https://support.microsoft.com/en-us/help/4021566/windows-10-send-feedback-to-microsoft-with-feedback-hub


Add tip about datetrunc use for start of month equivalent

Resubmitted using live branch - see #9470
I was looking for the "start of month" equivalent to eomonth and came across a tip to use datetrunc on stackoverflow. Hopefully this will help others looking for the same.
Let me know if any changes are needed, feel free to make edits or close!

@markingmyname
Can you review the proposed changes?
When the changes are ready for publication, add a #sign-off comment to signal that the PR is ready for the review team to merge.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

As I mentioned in your closed PR, this change has already been included in the refreshed article.
#please-close


Failed installing with pip install splunk-sdk for Python 3.12.0

Describe the bug
Install this with Python 3.12.0 via pip failed.
To Reproduce
Steps to reproduce the behavior:

Execute python -m pip install splunk-sdk, where python is Python 3.12.0

Expected behavior
It installs successfully.
Logs or Screenshots
(.venv) [jfcherng@3bc-f19 test]$ pip install splunk-sdk
Collecting splunk-sdk
  Using cached splunk-sdk-1.7.4.tar.gz (108 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [22 lines of output]
      Traceback (most recent call last):
        File "D:\jfcherng\Desktop\test\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "D:\jfcherng\Desktop\test\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\jfcherng\Desktop\test\.venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\jfcherng\AppData\Local\Temp\pip-build-env-cu0raqgr\overlay\Lib\site-packages\setuptools\build_meta.py", line 355, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\jfcherng\AppData\Local\Temp\pip-build-env-cu0raqgr\overlay\Lib\site-packages\setuptools\build_meta.py", line 325, in _get_build_requires
          self.run_setup()
        File "C:\Users\jfcherng\AppData\Local\Temp\pip-build-env-cu0raqgr\overlay\Lib\site-packages\setuptools\build_meta.py", line 507, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "C:\Users\jfcherng\AppData\Local\Temp\pip-build-env-cu0raqgr\overlay\Lib\site-packages\setuptools\build_meta.py", line 341, in run_setup
          exec(code, locals())
        File "<string>", line 22, in <module>
        File "C:\Users\jfcherng\AppData\Local\Temp\pip-install-lhx3336b\splunk-sdk_51a09a02018f4642aae48e2c7342d0c6\splunklib\__init__.py", line 18, in <module>
          from splunklib.six.moves import map
      ModuleNotFoundError: No module named 'splunklib.six.moves'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

SDK (please complete the following information):

Version:  1.7.4
Language Runtime Version: Python 3.12.0 amd64
OS: Win10 x64 21H2 19044.3448

May also related to https://github.com/splunk/splunk-sdk-python/issues/529

We are hitting the same issue in UCC framework, link from our pipeline (https://github.com/splunk/addonfactory-ucc-generator/actions/runs/6455317283/job/17522597650?pr=896#step:5:79).

Hi all, the library currently supports Python 2 and Python 3 runtimes and contains certain dependencies which are not working well with Python 3.9+ versions. This quarter, we are planning on a new beta release for the Python SDK with updated support for Python 3.9 and hope to resolve these issues. Please watch for updates here.

Is this still active? Still facing this issue.

I can confirm this is still an issue

Hi all, the library currently supports Python 2 and Python 3 runtimes and contains certain dependencies which are not working well with Python 3.9+ versions. This quarter, we are planning on a new beta release for the Python SDK with updated support for Python 3.9 and hope to resolve these issues. Please watch for updates here.

Q4 2023 is over. Is the support for 3.9+ version support still on the roadmap?

Is there a update on this?

Any updates on releasing the fix for this issue?

Are all version 3.9+ not supported or only 3.12? I have dependencies that only work on 3.10 and above, so a downgrade of python will cause other issues.

only 3.12 suffers.

Hello
I can confirm - issue is still present for python 3.12

issue is still present
❯ pip install splunk-sdk
Collecting splunk-sdk
Using cached splunk-sdk-1.7.4.tar.gz (108 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [22 lines of output]
Traceback (most recent call last):
File "/Users/P3186458/Documents/dev/CHARTER/SCP/out-of-home/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/Users/P3186458/Documents/dev/CHARTER/SCP/out-of-home/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/P3186458/Documents/dev/CHARTER/SCP/out-of-home/.venv/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/2r/_wwdbccn4pb3ckftlqfrx22r0000gp/T/pip-build-env-eu8a96eo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/private/var/folders/2r/_wwdbccn4pb3ckftlqfrx22r0000gp/T/pip-build-env-eu8a96eo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
self.run_setup()
File "/private/var/folders/2r/_wwdbccn4pb3ckftlqfrx22r0000gp/T/pip-build-env-eu8a96eo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 487, in run_setup
super().run_setup(setup_script=setup_script)
File "/private/var/folders/2r/_wwdbccn4pb3ckftlqfrx22r0000gp/T/pip-build-env-eu8a96eo/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
exec(code, locals())
File "", line 22, in
File "/private/var/folders/2r/_wwdbccn4pb3ckftlqfrx22r0000gp/T/pip-install-pfzeej4l/splunk-sdk_598c2b7d09bc40ffa272451a074aa307/splunklib/init.py", line 18, in
from splunklib.six.moves import map
ModuleNotFoundError: No module named 'splunklib.six.moves'
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.

Issue is still observed with python3.12.2

Hi all, Sorry for the delay, we have published a new version of Splunk Python SDK v2.0.0 with Python v3.7+ support . Request you to use the latest SDK and let us know if the issue is resolved. Thanks

Installation is okay. I will test our production codes to run with it tomorrow.

hi @ashah-splunk, I got an error when running with v2.0.0.
  File ".venv\Lib\site-packages\splunklib\results.py", line 49, in <module>
    import deprecation
ModuleNotFoundError: No module named 'deprecation'

After manually install deprecation from pypi, it works on my side.

Thanks @jfcherng for the fix PR. We will make sure to include the missing dependency in the next release.

Hi @jfcherng
We have published a patch release of Python SDK v2.0.1 with the missing 'deprecation' dependency.
Request you to upgrade to the latest version and let us know if any other issue pops up.
Thanks!

hi @ashah-splunk , v2.0.1 works well for me. Thanks.

Closed as my issue has been resolve.


bugfix: Fix sm75 kernel configuration

Some kernel configurations are not compatible with sm75, this pr fix these issues.

Ok I will verify it locally asap.

why disable sampling and cudagraph?

why disable sampling and cudagraph?

CUDA Graph doesn't work with Google Colab's T4 but works with GCP's T4, which is confusing. Also, Sampling doesn't work on sm75.
RuntimeError: TopKRenormProb failed with error code invalid device function

Sampling doesn't work on sm75.

I should fix this.

I reviewed the code and it should be compatible with sm75. I am going to extract this kernel and validate it on T4 separately.

Maybe we don't need to modify the const expr part subsequently, and choose to add --expt-relaxed-constexpr in the compilation parameters instead.

CUDA Graph doesn't work with Google Colab's T4 but works with GCP's T4, which is confusing.

I figured it out. This is because mem-frac was set when running on GCP T4, so actually CUDA Graph can be enabled. Since the VRAM of T4 is relatively small, the default mem-frac will prevent CUDA Graph from being enabled. This has nothing to do with FlashInfer. Cheers.

Maybe we don't need to modify the const expr part subsequently, and choose to add --expt-relaxed-constexpr in the compilation parameters instead.

The main issue is __CUDA_ARCH__ only works in device functions (inside kernels), and we can't rely on them in host functions.

Nice work! I will integrate it into SGLang.


need log of attempts

We can save this in a simple text file (~/.gemwarrior) for now, one line per attempt.

Game now logs some stats to a home directory file, and main menu has option to display it.


W-13845340: Update package-mule-application.adoc

sending for final review after publication to beta site


Minor grammar edit.

[x] I have read freeCodeCamp's contribution guidelines.
[x] My pull request has a descriptive title (not a vague title like Update index.md)
[ ] My pull request targets the master branch of freeCodeCamp.
[x] None of my changes are plagiarized from another source without proper attribution.
[x] My article does not contain shortened URLs or affiliate links.

If your pull request closes a GitHub issue, replace the XXXXX below with the issue number.
Closes #XXXXX

Thank you @vyrza for your contribution and congratulations on your first PR merge to this repo! 🎉


How to instrument code for Electron app?

Hi,
I am very sorry if this is actually not a bug, but I don't know any support channel and I am lost on this at the moment.
I wrote an Electron app, and setup a few tests with ava which start my app, trigger a few functions and terminate the application. nyc put on top only covers the executions from the Main Process.
As far as I understand, this is caused because nyc doesn't reach down to the browser context. Is there any way to build my entire Electron application with instrumented code, so I can still create a coverage report?
Here are a few snippets from my package.json:
"scripts": {
    "build": "npm run electron:serve-tsc && ng build --base-href ./",
    "ava-ts": "env NODE_ENV=ava ava-ts --serial",
},

  "ava-ts": {
    "cache": false,
    "serial": true,
    "verbose": true,
    "timeout": 10000,
    "instrument": true
  },

Any help is highly appreciated! Thank you!

Hi, is there any chance for support on this? Any help is appreciated! Thank you!

You likely need to use babel-plugin-istanbul, though I don't have any specific instructions for doing this in electron or under ava-ts.  This is a somewhat advanced topic which I don't have the knowledge to support.

@MMcKester, I wrote an article on how to setup TypeScript code coverage for Electron applications. Hope it helps you! 🍀


🐛 [firebase_auth ] PROVIDER_ID cannot be used in switch-case because it is not constant

Bug report
It is no longer possible to use AuthProvider.providerId nor AuthProvider.PROVIDER_ID inside switch-case statements because it is not a constant expression.
Also the question is why this was not written as a property at first because the current implementation is:
// google_auth.dart line 46

// ignore: public_member_api_docs
  static String get PROVIDER_ID {
    return _kProviderId;
  }

which is the same like:
  static String ROVIDER_ID = _kProviderId;

isnt it?
Steps to reproduce
// Before upgrade this worked
String providerId;
switch (providerId) {
    case GoogleAuthProvider.providerId:
        // do work
        break;
    case FacebookAuthProvider.providerId:
        // do work
        break;
}

// After upgrade cannot use switch-case anymore and have to refactor everything into if-elseif
String providerId;
switch (providerId) {
    case GoogleAuthProvider.PROVIDER_ID: //error: Case expressions must be constant.
        // do work
        break;
    case FacebookAuthProvider.PROVIDER_ID: //error: Case expressions must be constant.
        // do work
        break;
}

Expected behavior
switch case should work.

Additional context
Add any other context about the problem here.

Flutter doctor
Run flutter doctor and paste the output below:
Click To Expand
[√] Flutter (Channel stable, 1.20.4, on Microsoft Windows [Version 10.0.18363.1082], locale en-DE)
    • Flutter version 1.20.4 at D:\dev\flutter\sdk
    • Framework revision fba99f6cf9 (9 days ago), 2020-09-14 15:32:52 -0700
    • Engine revision d1bc06f032
    • Dart version 2.9.2

[!] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at C:\Users\User\AppData\Local\Android\sdk
    • Platform android-29, build-tools 29.0.3
    • Java binary at: C:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)
    X Android license status unknown.
      Try re-installing or updating your Android SDK Manager.
      See https://developer.android.com/studio/#downloads or visit https://flutter.dev/docs/get-started/install/windows#android-setup for detailed instructions.

[√] Android Studio (version 4.0)
    • Android Studio at C:\Program Files\Android\Android Studio
    • Flutter plugin version 49.0.2
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[!] VS Code (version 1.49.1)
    • VS Code at C:\Users\User\AppData\Local\Programs\Microsoft VS Code
    X Flutter extension not installed; install from
      https://marketplace.visualstudio.com/items?itemName=Dart-Code.flutter

[!] Connected device
    ! No devices available

! Doctor found issues in 3 categories.

@crtl
There's similar open issue describing your case, 3301.
Closing this as duplicate.


Update getting-started-with-group-managed-service-accounts.md

various smaller changes, dropping several WS12 and TechNet references. Note: Uninstall-ADServiceAccount link may need work.

@robinharwood, @Xelu86
Can you review the proposed changes?
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team


Specified versions of node and golang aren't available in buildpacks

Remove specification for node version and just use whatever is
available
Bump to Golang 1.8

[#146554555]
Signed-off-by: Corey Innis<EMAIL_ADDRESS>

Hey @SocalNick, could you take a look at the Travis failure?

@dsabeti I removed ruby 1.9.3 as it was EOL in Feb 2015

@SocalNick sounds too reasonable to me.


Unit tests for high level models very sparse

Even though we've got 100% code coverage, the unit tests for the high level models, e.g. partners and teacherSearch, aren't currently testing a lot of the models' functionality. Additional unit tests should be added to better validate the model code.

Looking into this now.

Stephen, just wanted to let you know that I'm back to focusing on this issue. I'll continue to familiarize myself with the tests and models. If there is a higher priority task that you need completed, let me know!
Thanks.

Turns out that there wasn't a lot more to add. I've taken care of this


Suggested alternative approach

I came across this plugin but it didn't quite solve my desires. However, it was a helpful starting point, and I think the results I got might be useful to share here (either to incorporate or for others to copy).
This will automatically grab a user's existing configuration, and toggle between that config and everything off (in contrast to this plugin currently, which ignores the existing config).
DiagnosticsConfig = vim.diagnostic.config()
DiagnosticsEnabled = true
function ToggleDiagnostics()
   if not DiagnosticsEnabled then
      vim.diagnostic.config(DiagnosticsConfig)
      DiagnosticsEnabled = true
   else
      DiagnosticsConfig = vim.diagnostic.config()
      vim.diagnostic.config({
         virtual_text = false,
         sign = false,
         float = false,
         update_in_insert = false,
         severity_sort = false,
         underline = false,
      })
      DiagnosticsEnabled = false
   end
end
vim.keymap.set("n", "yod", ToggleDiagnostics, { noremap=true, silent=true })

Thanks. I will look into being more tolerant of current configuration.

fwiw, this plugin's init() routine allows passing the same things you pass to vim.diagnostic.config(). The idea being that you only call this plugin's init(). However I see the value in preserving the current config.

Thank you for your example code. While toggle-lsp-diagnostics will continue to turn everything on by default (as the documentation promises) you can pass in the current configuration to init() like so:
require'toggle-lsp-diagnostics'.init(vim.diagnostic.config())

This will have the affect you get with your code snippet. Thanks again.

I think the issue is that your toggle doesn't return the behavior to what was specified in the init (as far as I can tell). I think generally people expect a toggle to change between two states, but currently you change between 3 - the current/default config, all on, or all off. It should toggle by default between all off or whatever was set by the init. IMO you should store the init


Array mutation without using helper methods

What's the recommended way to signal a change on an array when we're not using the helper methods? For instance, if I want to sort an array, what's the best way to let Polymer know something has changed? At the moment I'm using this.set('myArray', myArray.slice(0)); but is there a better way?

Note that dom-repeat supports filtering and sorting the view, such that for many use cases you don't need to mutate (sort/filter) the actual underlying array: https://www.polymer-project.org/1.0/docs/devguide/templates.html#filtering-and-sorting-lists
Right now, Polymer's array observation doesn't support observing in-place array sorts, so the workaround you are using is probably the best option at the moment if you absolutely need to sort the actual array for some app-specific reason.


fix (proposed): set the default tfenv to linux for EC2 instances, local dev, etc.

I'm not sure if this is the smartest change, but I think it could be a first step. For a lot of us with M1 macs, we run into incompatibilities where our tfenv architecuture is set to darwin_arm64 instead of the linux one. Most of our EC2 Instances run in amd though--we don't want to change the default to the newest Macbook default.
For example, I'd get this error a lot when I run terraform locally:
╷
│ Error: Incompatible provider version
│
│ Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a
│ package available for your current platform, darwin_arm64.
│
│ Provider releases are separate from Terraform CLI releases, so not all
│ providers are available for all platforms. Other versions of this provider
│ may have different platforms supported.
╵

I couldn't get rid of it until I modified the conditions of the tf_env install--I didn't care about getting the M1 Macbook version so I looked into ways to convince tf_env I'm using linux_amd64 architecture.
This change exports and sets a TFENV architecture to Linux by default--I have not tested whether we can override it back to arm64 or not.

I think the real problem here is that some of our old modules utilized registry.terraform.io/hashicorp/template. I actually fixed it so that the module is updated, but some workspaces still reference the old one. Their state's need to be refreshed with the latest module before that disappears entirely. Here's the PR that I used to fix it:
https://github.com/chanzuckerberg/shared-infra/pull/7716

I'm confused about the comment about EC2 instances. Why would that matter in this case?


PEP8

Updating for PEP8 standards.  Using the .format() string built-in over the %s style.  Also, removed unused imports.
No functional changes were made in this pull request.  This "clean up" was made while reviewing the source code to see where I can contribute.

I hate to be a downer, but like Django I don't follow the 80-character lines part of PEP8, so I'm not going to accept this with all the line break changes - it makes the code a lot harder to read.
If you want to undo the line breaks but keep .format() etc, I'll happily merge those.

I'll undo the 80-character breaks (although I strongly disagree :P).
I'd love to see this project get to 1.0 as quickly as possible.  Please let me know if there is anything you'd like me to work on (functionally).  I've got a few months of free time and this project is now my number 1 priority.  :)  rico.cordova at gmail.com

I guess the code should follow the django coding style (https://docs.djangoproject.com/en/1.9/internals/contributing/writing-code/coding-style/)
I don't think replacing the format() stuff is necessary ( since under python3 the % operator for strings is twice as fast ) and it's not deprecated.

Closing for now, outdated - reopen if you get to a new version.


OpenSourceApp Invalid model, Unable to instantiate model

🐛 Describe the bug
Documenting for others, not sure if there is something for embedchain to "fix" or not.
Basic, initial install ubuntu 22.04, python 3.10
pip install embedchain[opensource] went fine.
from embedchain import OpenSourceApp
app = OpenSourceApp()

Failed with the following traceback:
ValueError                                Traceback (most recent call last)
Cell In[4], line 2
      1 from embedchain import OpenSourceApp
----> 2 app = OpenSourceApp()

File /mnt/c/Users/Blake/pyai/embedchain/lib/python3.10/site-packages/embedchain/apps/OpenSourceApp.py:36, in OpenSourceApp.__init__(self, config, system_prompt)
     33 if not config.model:
     34     raise ValueError("OpenSourceApp needs a model to be instantiated. Maybe you passed the wrong config type?")
---> 36 self.instance = OpenSourceApp._get_instance(config.model)
     38 logging.info("Successfully loaded open source embedding model.")
     39 super().__init__(config, system_prompt)

File /mnt/c/Users/Blake/pyai/embedchain/lib/python3.10/site-packages/embedchain/apps/OpenSourceApp.py:53, in OpenSourceApp._get_instance(model)
     48 except ModuleNotFoundError:
     49     raise ModuleNotFoundError(
     50         "The GPT4All python package is not installed. Please install it with `pip install embedchain[opensource]`"  # noqa E501
     51     ) from None
---> 53 return GPT4All(model)

File /mnt/c/Users/Blake/pyai/embedchain/lib/python3.10/site-packages/gpt4all/gpt4all.py:98, in GPT4All.__init__(self, model_name, model_path, model_type, allow_download, n_threads, device)
     96     if device != "cpu":
     97         self.model.init_gpu(model_path=self.config["path"], device=device)
---> 98 self.model.load_model(self.config["path"])
     99 # Set n_threads
    100 if n_threads is not None:

File /mnt/c/Users/Blake/pyai/embedchain/lib/python3.10/site-packages/gpt4all/pyllmodel.py:267, in LLModel.load_model(self, model_path)
    265     llmodel.llmodel_loadModel(self.model, model_path_enc)
    266 else:
--> 267     raise ValueError("Unable to instantiate model")
    269 filename = os.path.basename(model_path)
    270 self.model_name = os.path.splitext(filename)[0]

ValueError: Unable to instantiate model

I confirmed the model downloaded correctly and the md5sum matched the gpt4all site.
Downgrading gtp4all to 1.0.8 fixed the issue.
pip install --force-reinstall -v "gpt4all==1.0.8"
Found the solution from this thread: https://github.com/nomic-ai/gpt4all/issues/1394

Yes, I had the same problem (posted about it on discord), but thought it's a me issue. We will raise a PR to pin the gpt4all version.

My error was OSError: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.29' not found (for context)


Proxy persist proxy ports 1.15

[ ] #32973 (@jrajahalme)

Once this PR is merged, a GitHub action will update the labels of these PRs:
 32973

/test-backport-1.15

@jrajahalme I think you also need to backport https://github.com/cilium/cilium/pull/33230


0.3.4 fix

Note: googlebot is wrong about the CLA.

cc9789d4470cc4c93a26c61e5ea5a784cbb2fc90 is a rather separate issue (which, incidentally, I personally support, but I'm just an assistant here, @shinh and @ukai really run this show), I'd pull that out in a separate PR.

Closing due to missing CLA.


Results are not being saved to testrail

Node: 12.16.3
Jest: 26.0.1
This is the config

module.exports = {
verbose: true,
moduleFileExtensions: ["js", "json", "ts"],
globals: {
'ts-jest': {
tsconfig: {
target: 'ESNext',
module: 'commonjs',
strict: true,
esModuleInterop: true,
forceConsistentCasingInFileNames: true,
types: ['@types/jest', 'jest-playwright-preset', 'expect-playwright']
}
}
},
preset: 'jest-playwright-preset',
transform: {'^.+\.ts$': 'ts-jest'},
testMatch: ['**/playwright/tests/*.test.ts'],
reporters: [
'default',
[
'jest-junit',
{
outputDirectory: './test-reports',
outputName: 'results.xml',
usePathForSuiteName: true,
classNameTemplate: '{classname}-{title}',
titleTemplate: '{classname}-{title}'
}
],
["jest-2-testrail", { project_id: "1", suite_id: "1" }]
],
testRunner: 'jest-circus/runner',
testEnvironment: './customEnvironment.js',
testTimeout: 50000,
// will be loaded soon per default in jest-playwright-preset
setupFilesAfterEnv: ['expect-playwright']
};

The result is
Test Suites: 2 failed, 2 total
[1] Tests:       2 failed, 2 total
[1] Snapshots:   0 total
[1] Time:        90.239 s
[1] Ran all test suites.
[1] Created new test run: Master - May 27, 2020, 16:22:59 - (Automated test run)
[1] Field :results cannot be empty (one result is required)

@vrknetha were u able to figure out? the error? I am running into same issue, it did create a test run for me but cannot save the results of the test cases

@vrknetha You need to append only numeric in the test case ID and not with the alphabet. in my case, I changed it from C1 to just 1 and it started behaving fine


Fix strfmt version.

Description
The strfmt version defined in the go.mod file is invalid. This commit fixes that by defining a valid version.

does this need a release?

does this need a release?

I think so. We need to update the v1.24.0-kw3 tag

Yes we do. Can you please handle that once you merge this PR?

does this need a release?

I think so. We need to update the v1.24.0-kw3 tag

Yes we do. Can you please handle that once you merge this PR?

Yes. That's in my to do list already. ;)


[impv] Only create one connect for single workflow submit

Brief Summary of The Change

we directly use class JavaGate to create py4j connection in
each place we want to connect Java gateway. It will make java gateway
to many connections, this patch try to init the java_gateway instance
only one time during the workflow submit, to save the cause during
connection create
Pull Request checklist
I confirm that the following checklist has been completed.

[x] Add/Change test cases for the changes.
[x] Add/Change the related documentation, should also change docs/source/config.rst when you change file default_config.yaml.
[x] (Optional) Add your change to UPDATING.md when it is an incompatible change.

Codecov Report

Merging #16 (0600be3) into main (0b7c0be) will not change coverage.
The diff coverage is 31.25%.

@@           Coverage Diff           @@
##             main      #16   +/-   ##
=======================================
  Coverage   90.59%   90.59%
=======================================
  Files          56       56
  Lines        2042     2042
=======================================
  Hits         1850     1850
  Misses        192      192

Flag
Coverage Δ

unittests
90.59% <31.25%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

src/pydolphinscheduler/models/project.py
58.33% <20.00%> (ø)

src/pydolphinscheduler/models/tenant.py
56.66% <20.00%> (ø)

src/pydolphinscheduler/models/user.py
50.00% <20.00%> (ø)

src/pydolphinscheduler/core/process_definition.py
81.86% <33.33%> (ø)

src/pydolphinscheduler/core/resource.py
86.36% <33.33%> (ø)

src/pydolphinscheduler/core/task.py
93.67% <33.33%> (ø)

src/pydolphinscheduler/core/database.py
73.07% <50.00%> (ø)

src/pydolphinscheduler/core/engine.py
78.78% <50.00%> (ø)

src/pydolphinscheduler/tasks/dependent.py
94.57% <50.00%> (ø)

src/pydolphinscheduler/tasks/sub_process.py
83.33% <50.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Problems with Celery

Environment

Nautobot version (Docker tag too if applicable): nautobot:stable-py3.11
Python version: 3.11
Database platform, version: postgres:13
Middleware(s): default

Steps to Reproduce

I have create this job in the root folder for jobs (/opt/nautobot/jobs:

## Tutorial: https://josh-v.com/nautobot-jobs-in-jobs-root/
from django.conf import settings

from nautobot.dcim.models import Device

from nautobot.apps.jobs import Job, ObjectVar, register_jobs

from netmiko import ConnectHandler

name = "Get Version Huawei Smartax EA5800"

class GetVersionOLT(Job):
    device = ObjectVar(
        model=Device,
        description="Device to get the version of"
    )

    def run(self, device):
        self.log_debug(device)

    class Meta:
        name = "Get Version OLT"
        description = "Get the version of the OLT"

register_jobs(GetVersionOLT)

I do the nautobot-server post_upgrade and restart the dockers for nautobot, celery_worker and celery_beat
I try to run the job create

Expected Behavior
It should run the job without any major issue, printing the data from the device.
Observed Behavior
It crashed the celery_worker saying the following:
2024-03-08 14:54:08 [2024-03-08 13:54:08,522: WARNING/MainProcess] /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
2024-03-08 14:54:08 whether broker connection retries are made during startup in Celery 6.0 and above.
2024-03-08 14:54:08 If you wish to retain the existing behavior for retrying connections on startup,
2024-03-08 14:54:08 you should set broker_connection_retry_on_startup to True.
2024-03-08 14:54:08   warnings.warn(
2024-03-08 14:54:08
2024-03-08 14:54:08 [2024-03-08 13:54:08,536: WARNING/MainProcess] /usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
2024-03-08 14:54:08 whether broker connection retries are made during startup in Celery 6.0 and above.
2024-03-08 14:54:08 If you wish to retain the existing behavior for retrying connections on startup,
2024-03-08 14:54:08 you should set broker_connection_retry_on_startup to True.
2024-03-08 14:54:08   warnings.warn(
2024-03-08 14:54:08
2024-03-08 14:55:11 [2024-03-08 13:55:11,472: ERROR/MainProcess] Received unregistered task of type 'get_version_olt_jobs.GetVersionOLT'.
2024-03-08 14:55:11 The message has been ignored and discarded.
2024-03-08 14:55:11
2024-03-08 14:55:11 Did you remember to import the module containing this task?
2024-03-08 14:55:11 Or maybe you're using relative imports?
2024-03-08 14:55:11
2024-03-08 14:55:11 Please see
2024-03-08 14:55:11 https://docs.celeryq.dev/en/latest/internals/protocol.html
2024-03-08 14:55:11 for more information.
2024-03-08 14:55:11
2024-03-08 14:55:11 The full contents of the message body was:
2024-03-08 14:55:11 b'[[], {"device": "c2c05bf8-bd96-4c5c-8545-425504323d35"}, {"callbacks": null, "errbacks": null, "chain": null, "chord": null}]' (125b)
2024-03-08 14:55:11
2024-03-08 14:55:11 The full contents of the message headers:
2024-03-08 14:55:11 {'lang': 'py', 'task': 'get_version_olt_jobs.GetVersionOLT', 'id': '7aaae6da-91af-414e-8a48-6d2b3eb20699', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [0, 0], 'root_id': '7aaae6da-91af-414e-8a48-6d2b3eb20699', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': "{'device': UUID('c2c05bf8-bd96-4c5c-8545-425504323d35')}", 'origin': 'gen74@51696c90f8fb', 'ignore_result': False, 'replaced_task_nesting': 0, 'stamped_headers': None, 'stamps': {}}
2024-03-08 14:55:11
2024-03-08 14:55:11 The delivery info for this task is:
2024-03-08 14:55:11 {'exchange': '', 'routing_key': 'default'}
2024-03-08 14:55:11 Traceback (most recent call last):
2024-03-08 14:55:11   File "/usr/local/lib/python3.11/site-packages/celery/worker/consumer/consumer.py", line 658, in on_task_received
2024-03-08 14:55:11     strategy = strategies[type_]
2024-03-08 14:55:11                ~~~~~~~~~~^^^^^^^
2024-03-08 14:55:11 KeyError: 'get_version_olt_jobs.GetVersionOLT'
2024-03-08 14:55:11 [2024-03-08 13:55:11,475: CRITICAL/MainProcess] Unrecoverable error: AttributeError("'Bunch' object has no attribute 'delivery_info'")

Is /opt/nautobot/jobs/get_version_olt_jobs present in both the server container and the celery worker container?

ooohh I didn't add it in the celery worker. I didn't see that in the job's documentation. I will check it out on monday :)

Hi! I'm sorry for the problem. Yes, it was simply the celery worker container problem :( .
Thanks a lot for the reply!