Running error: gci: failed to diff: exec: "diff": executable file not found in $PATH

Hi,
I'm using a custom docker files, that does not contain a diff tool, and the golangci-lint fails with an error:
level=warning msg="[runner] Can't run linter goanalysis_metalinter: gci: failed to diff: exec: \"diff\": executable file not found in $PATH"
level=error msg="Running error: gci: failed to diff: exec: \"diff\": executable file not found in $PATH"

Please get rid of usage the diff tool, please use the github.com/shazow/go-diff/difflib library, example of ho to use it you can find here: https://github.com/golangci/golangci-lint/blob/master/pkg/golinters/gofumpt.go#L62

Instead of this: https://github.com/daixiang0/gci/blob/master/pkg/gci/gci.go#L178-L198
you can do like this:
	out := bytes.Buffer{}
	_, err = out.WriteString(fmt.Sprintf("--- %[1]s\n+++ %[1]s\n", f))
	if err != nil {
		return nil, fmt.Errorf("error while running gofumpt: %w", err)
	}
	err = differ.Diff(&out, bytes.NewReader(b1), bytes.NewReader(b2))
	if err != nil {
		return nil, fmt.Errorf("error while running gofumpt: %w", err)
	}
	return out.Bytes()

and least it will be faster, then using a subcommand

Goimports still call diff, need to dig deeper.

I would prefer to turn off goimports than gci  :)

Thanks!
While I do not want any extra pkg here, so i would update gci.Run and the way to call in golang-ci.

thank you for the quick fix!!!!


Is the current group (about.k8s.io) for clusterproperty API accurate?

The clusterproperty API group is currently defined as group: about.k8s.io
https://github.com/kubernetes-sigs/about-api/blob/master/clusterproperty/config/crd/bases/about.k8s.io_clusterproperties.yaml#L10
But as stated in KEP k8s.io group protection https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/2337-k8s.io-group-protection/README.md
This group should be protected unless there is an annotation with the PR attached.
The current kustomize patch annotation points to a not related API (default template value?): https://github.com/kubernetes/kubernetes/pull/78458
https://github.com/kubernetes-sigs/about-api/blob/master/clusterproperty/config/crd/patches/annotation_for_clusterproperties.yaml#L6
Should the clusterproperty API group be x-k8s.io instead?

related ^
go get github.com/kubernetes-sigs/about-api/clusterproperty@f06d7db18c74a8da69510b755bad4e6f55679b14
go: downloading github.com/kubernetes-sigs/about-api/clusterproperty v0.0.0-20220614182201-f06d7db18c74
go: downloading github.com/kubernetes-sigs/about-api v0.0.0-20220614182201-f06d7db18c74
go get<EMAIL_ADDRESS>parsing go.mod:
	module declares its path as: k8s.io/clusterproperty
	        but was required as: github.com/kubernetes-sigs/about-api/clusterproperty

about.k8s.io is correct, but the patch annotation is the wrong link. Our API approval PR is https://github.com/kubernetes/enhancements/pull/3084


Printing of "Loading required atoms [:iat, :aud, :sub, :exp, :iss, :pems]"

This is really intrusive into my sea of green dots while running my tests. Is this still necessary? If so, I am quite curious as to why this is required.

@bbhoss this was part of a joken thing which I'm now removing. Making it so that keys are strings not atoms. I will at the same time remove that print

This has been removed in Guardian 0.5.0


Change to buster?

We now have distroless builds for buster out! We have this set so far:
static-debian11:latest-amd64
static-debian11:nonroot-amd64
static-debian11:debug-amd64
static-debian11:debug-nonroot-amd64
static-debian11:latest-arm64
static-debian11:nonroot-arm64
static-debian11:debug-arm64
static-debian11:debug-nonroot-arm64
static-debian11:latest-arm
static-debian11:nonroot-arm
static-debian11:debug-arm
static-debian11:debug-nonroot-arm
static-debian11:latest-s390x
static-debian11:nonroot-s390x
static-debian11:debug-s390x
static-debian11:debug-nonroot-s390x
static-debian11:latest-ppc64le
static-debian11:nonroot-ppc64le
static-debian11:debug-ppc64le
static-debian11:debug-nonroot-ppc64le
base-debian11:latest-amd64
base-debian11:nonroot-amd64
base-debian11:debug-amd64
base-debian11:debug-nonroot-amd64
base-debian11:latest-arm64
base-debian11:nonroot-arm64
base-debian11:debug-arm64
base-debian11:debug-nonroot-arm64
base-debian11:latest-arm
base-debian11:nonroot-arm
base-debian11:debug-arm
base-debian11:debug-nonroot-arm
base-debian11:latest-s390x
base-debian11:nonroot-s390x
base-debian11:debug-s390x
base-debian11:debug-nonroot-s390x
base-debian11:latest-ppc64le
base-debian11:nonroot-ppc64le
base-debian11:debug-ppc64le
base-debian11:debug-nonroot-ppc64le

I think that's enough for ko, should we start testing and switch the default?

Why on earth would you name them like this?

Which part? None of this is new really. There were 10s everywhere that there are now 11s. I think a bunch of these are also multi-platform images.

None of the ones in that list are multi-platform. I'd drop all the platform-specific tags and just have multi-platform images only.
Encoding nonroot+debug in the tag is dumb, can those be in the repository path instead?

Encoding nonroot+debug in the tag is dumb, can those be in the repository path instead?

PRs welcome!

None of the ones in that list are multi-platform. I'd drop all the platform-specific tags and just have multi-platform images only.

It doesn't work that way. Looks like we need to update this script to add the multi-arch 11's too: https://github.com/GoogleContainerTools/distroless/blob/main/cloudbuild_docker.sh#L28

Naming aside, this begs the question of how we should roll out changes to the base image ko uses by default. In #160 we switched from rootful to :nonroot distroless, with a warning and a release before switching the default. Should we do that again for buster (and presumably 12, 13, etc., as they come out)? That seems pretty choresome, especially when we expect users not to care in most cases.
Can distroless just change its definition of gcr.io/distroless/static:nonroot to point to buster, then 12, 13, etc.? If users want to pin to a specific Debian version they can do so with -debian11 etc., and ko just wouldn't by default.

Ref https://github.com/GoogleContainerTools/distroless/issues/876
It looks like ko still uses debian10. From some logs today on ko 0.9.3:
2021/10/25 00:04:44 Using base gcr.io/distroless/base-debian10 for github.com/sigstore/fulcio

It looks like distroless/static:nonroot is already bullseye:
→ crane export gcr.io/distroless/static:nonroot - | tar -xO ./usr/lib/os-release
PRETTY_NAME="Distroless"
NAME="Debian GNU/Linux"
ID="debian"
VERSION_ID="11"
VERSION="Debian GNU/Linux 11 (bullseye)"
HOME_URL="https://github.com/GoogleContainerTools/distroless"
SUPPORT_URL="https://github.com/GoogleContainerTools/distroless/blob/master/README.md"
BUG_REPORT_URL="https://github.com/GoogleContainerTools/distroless/issues/new"

Indeed, a ko-built image (built a while ago) is based on bullseye:
$ crane export gcr.io/tekton-nightly/github.com/tektoncd/pipeline/cmd/entrypoint@sha256:7917ba5196b1a43058452ff93cd47e7255a73d4a45e56c7810d3f03e2e15e693 - | tar -xO ./usr/lib/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"


enhancement: sankey diagram visualization

I was excited to see gandtt charts added in the recent release.
Another great addition to the list of available visualizations would be a sankey diagram. Here are some references:

Wikipedia
Sankey diagrams in addition to Elastic.co's non-open-source Kibana in Vega
Community plugin on GitHub (works up through Elastic 7.6.2 but not in new framework)

Please do what Elastic company can't do for a long time. The whole world is waiting for fantasty - nested field support for kibana.


extension does not loaded properly in 10.13 high sierra

https://github.com/tekezo/Karabiner-Elements/issues/777

@jostyee thanks for the link, close as duplicate


connect with factorsDB

Connect the app with factorsdb.org database: http://www.factorsdb.org/
This will serve as a repository for experiments.
https://github.com/jodeleeuw/factorsdb

one issue with using this database is that the experiments will not have triggers implemented.. ill have to touch base with @jodeleeuw on this

I am going to need to connect my app to a db as well, so i will pay close attention here!


Update UTM campaign

Pull Request
What does this PR do?
Redirect CTAs to Cloud landing page

@gillian-meilisearch do not merge wtihout the @Strift approval, I want to be sure we are ok with the analytics convention 😊


tsp, support string list and integer list as paged response

case
/**
 * The list of strings with server paging support.
 */
@Azure.Core.pagedResult
model StringsList {
  /** The collection of pageable items. */
  @Azure.Core.items
  value: string[];

  /** The link to the next page of items */
  @Azure.Core.nextLink
  nextLink?: string;
}

/**
 * The list of int32s with server paging support.
 */
@Azure.Core.pagedResult
model Int32sList {
  /** The collection of pageable items. */
  @Azure.Core.items
  value: int32[];

  /** The link to the next page of items */
  @Azure.Core.nextLink
  nextLink?: string;
}

There may be e.g. paged utc-data-time though...


Refactor carousel

This PR refactors the carousel to un-nest CSS selectors, simplify overall styling, and improve documentation of key features. More specifically:

New visuals for a simpler design. I've removed the gradients on the controls, changed the indicators from circles to squares (with larger tap areas), and added new icons.

Un-nest CSS. Instead of .carousel > .item and the like, we have .carousel-* prefixed on all classes (save for the .active state). This includes the slides, controls, and indicators. Fixes #18396.

Simply CSS for sliding effects. We added transforms later in v3's lifecycle, so we had to support transitions for position and transform. No more; now we just have 3D transforms, so our compiled CSS is much simpler.

More reliance on utilities. Image styles for within carousel items have been removed, so now you'll need to use utilities or custom styles (e.g., .d-block and .img-fluid). Similarly, I've included an example of responsive captions in our docs built with utilities.

Improved documentation. The original example is now split across a handful of examples, each building on the previous (from slides only, to controls, to indicators, to captions). Rewrites some of the intro docs and also provides better guidance on responsive captions and more.

Tweak JS to better handle directional states. This pulls in the JS changes from #18830 to accomplish that.

Fixes #17878, correcting the Carousel example.

Update migration docs to mention changes.

Pushed a ton more updates to this since opening the PR. The only remaining thing this doesn't address yet is a flexbox variant. I'll tackle that tomorrow I think. Things that might benefit from flexbox:

Control-caption dimensions staying in sync
Control icon alignment
Indicator alignment

Some of this likely can be done with utilities, too.

Hello, I was briefly looking at the code change and just happened to find a small typo in the migration.md file at line 190. The new text in the PR has duplicate .carousel-control-left as shown below
For prev/next controls, `.carousel-control.right` and `.carousel-control.left`
are now `.carousel-control-left` and `.carousel-control-left`,
meaning they no longer require a specific base class.

Thanks for this great framework :)

Punting flexbox variant to another PR. Opened #21305 to track.


feat: improve format and time interval guidance (#641)

fixes #641.
Improves the guidance on time interval and duration usage and adds some small changes we have detected over the last month where we have to be more specific about the usage of (data-)formats in the guideline.
Also fixes some small etag property presentation issue.

👍

:+1:


Ux updates 7

A series of UX updates that includes the following:

Put the instructional bursts on the left column, the widgets on the right.
Switching the intro slideshow GET STARTED button so that it is for internal use and will be deleted except on the final slide.
Delete the helper alerts at the bottom of the page
Restyle the Messier Buttons, as well as the Pan/Zoom instructions.

There's some inconsistent behavior of the Messier object buttons on slide 4 vs 5 on my computer


Liveness probes busted

Pods should be restarted with a /healthz http liveness probe (Failed 5 times in the last 9 runs. Stability: 44 %)
Pods should be restarted with a docker exec "cat /tmp/health" liveness probe (Failed 3 times in the last 9 runs. Stability: 66 %)
This test was ported in #4280. However, as much as I wanted to blame that hunk of code (because the stability of TestLivenessExec and TestLivenessHttp were pretty strong), even after reverting #4280, I still liveness probes busted.
As best I can tell, these went unstable sometime around v0.10.0-419-g7d43971 (this is the git describe in the build for job/kubernetes-e2e-gce/2158/  on our jenkins server, and before #4280 went in). Unfortunately, because it's a flake, though a really solid one, it could also be v0.10.0-409-g371349f. Seems likely that it was somewhere in that vicinity, though.
cc @brendanburns
This is basically P0, because it wrecks our CI.

cc @mikedanese

I noticed that in at least one of the failed runs, we tried to join the
network container of the pod that we had just killed.  There's something
weird going on, as we shouldn't even be killing the network container in
the first place.  I'm going to dig in a little more on this...
(And yes, the readiness refactor is a potential cause, will dig more...)
On Feb 12, 2015 7:24 PM, "Brian Grant"<EMAIL_ADDRESS>wrote:

cc @mikedanese https://github.com/mikedanese
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/issues/4406#issuecomment-74199278
.

Also be aware of #4462 which can lead to instability due to read races in the Kubelet, even if #4410 was the proximate cause.

Actually, there's something fishy here .. it's possibly the Go test (but it has nothing to do with @rrati's port). I just noticed we still have liveness.sh in the tree, and it's doing fine: Shell tests that liveness.sh passes (Failed 1 times in the last 100 runs. Stability: 99 %)

I believe this is resolved.


Extract (private) functionality of ServerRequestFactory to functions

This patch serves as an alternative to #306. It extracts a number of methods from ServerRequestFactory into dedicated functions, allowing them to be used to derive ServerRequest artifacts. They include:

Uploaded file creation:

CREATES:

Zend\Diactoros\createUploadedFile(array $spec) : UploadedFile
Zend\Diactoros\normalizeUploadedFiles(array $files) : UploadedFileInterface[]
Zend\Diactoros\normalizeUploadedFileSpecification(array $files) : UploadedFile[]

REMOVES:

ServerRequestFactory::createUploadedFileFromSpec()
ServerRequestFactory::normalizeNestedFileSpec()

DEPRECATES:

ServerRequestFactory::normalizeFiles(); the method is no longer used internally, and now proxies to normalizeUploadedFiles().

Protocol version detection:

CREATES Zend\Diactoros\marshalProtocolVersion(array $server) : string
REMOVES ServerRequestFactory::marshalProtocolVersion()

Cookie header parsing:

CREATES Zend\Diactoros\parseCookieHeader(string $cookieHeader) : array
REMOVES ServerRequestFactory::parseCookieHeader()

Host and port discovery:

ADDS:

Zend\Diactoros\getHeaderFromArray(string $name, array $headers, $default = null), for extracting a value from a header array.
Zend\Diactoros\marshalHostAndPort(array $headers, array $server) : array, to allow detecting the host and port from either the request headers or the SAPI environment.

DEPRECATES:

ServerRequestFactory::getHeader(), and has it proxy to getHeaderFromArray().
ServerRequestFactory::marshalHostAndPortFromHeaders(), and has it proxy to `marshalHostAndPort(). It is no longer used internally.

REMOVES:

ServerRequestFactory::marshalHostAndPortFromHeader()
ServerRequestFactory::marshalIpv6HostAndPort()

$_SERVER normalization:

ADDS Zend\Diactoros\normalizeServer(array $server, callable $apacheRequestHeaderCallback = null) : array) for preparing the SAPI params to include the Authorization header; this is mainly done to ensure it is extracted properly under Apache.
DEPRECATES ServerRequestFactory::normalizeServer(), and has it proxy to normalizeServer(); it is no longer used internally.

Header marshaling:

ADDS Zend\Diactoros\marshalHeadersFromSapi(array $server) : array for marshaling HTTP request headers from the SAPI $_SERVER array.
DEPRECATES ServerRequestFactory::marshalHeaders(), and has it proxy to marshalHeadersFromSapi(); the method is no longer used internally.

Path/URI marshaling:

ADDS:

Zend\Diactoros\marshalRequestPath(array $server) : string for marshaling the request path from the SAPI parameters.
Zend\Diactoros\marshalUriFromServer(array $server, array $headers) : Uri for marshaling a Uri instance from the SAPI parameters and/or HTTP request headers. The function relies on getHeaderFromArray(), marshalHostAndPort(), and marshalRequestPath(), and inlines logic from ServerRequestFactory::stripQueryString().

DEPRECATES:

ServerRequestFactory::marshalRequestUri(), and has it proxy to marshalRequestPath(). The method is no longer used internally.
ServerRequestFactory::marshalUriFromServer(), and has it proxy to marshalUriFromSapi() The method is no longer used internally.
ServerRequestFactory::stripQueryString(). The method is no longer used internally.

Outstanding questions:

I have some new function use the FromSapi() suffix, while others do not. Add it to those that do not, or remove it from those that do?
I use the prefix normalize with two functions that aggregate UploadedFileInterface instances; would it be better to use the marshal prefix as I do elsewhere?

Thanks, @Moln ; incorporated both in that method, as well as the inlined version in marshalUriFromSapi().

I've made the following changes at this time:

Consolidated all URI generation functions into a single Zend\Diactoros\marshalUriFromSapi() function; this allowed removing each of getHeaderFromArray(), marshalHostandPort(), and marshalIpv6HostAndPort().
Renamed the protocol version detection function to Zend\Diactoros\marshalProtocolVersionFromSapi().
Added a method detection function, Zend\Diactoros\marshalMethodFromSapi(array $server) : string.
Consolidated the upload file normalization functions into Zend\Diactoros\normalizeUploadedFiles(); createUploadedFile() was kept, as it has standalone use cases.

I've also updated ServerRequestFactory to follow all changes.
At this point, I think this is ready for final review.


Question about default styles.

How to delete the default styles?
I get that I can overwrite them with the syntax provided in the documentation, but it's a bit problematic when using default style sheets for "input" cause I don't have necessarily have written the css by myself, specially using tailwinds, so I need to figure out what is the style and that one can also dynamically change (dark-mode etc).
Of course i can modify the code and delete the styles, but I was wondering if there is any other option?

@PabloRuizCuevas just pushed an update that allows you to completely override the CSS class names so you don't have to write your own CSS.
I'm not super familiar with Tailwind or how exactly you are achieving Dark mode but let me know if the latest changes fix the issue for you.
Updated the docs with instructions on how to use it.

@fmaclen  interesting,I test it, thanks!


Question - Does Nancy FX support multi threading?

More of a question than a issue. Does Nancy FX support multi threading?

What do you mean by supporting multi-threading? Supporting concurrent
requests?  Supporting modules that process on multiple threads?  Supporting
multiple instances of a Nancy server in different threads...?  Something
else entirely?
On Nov 15, 2017 8:47 AM, "Mohamed Shazan"<EMAIL_ADDRESS>wrote:

Prerequisites

I have written a descriptive issue title
I have verified that I am running the latest version of Nancy
I have verified if the problem exist in both DEBUG and RELEASE mode
I have searched open https://github.com/NancyFx/Nancy/issues and
closed
https://github.com/NancyFx/Nancy/issues?q=is%3Aissue+is%3Aclosed
issues to ensure it has not already been reported

Description
More of a question than a issue. Does Nancy FX support multi threading?
Steps to Reproduce System Configuration

Nancy version:
Nancy host

Nancy.Hosting.Aspnet
Nancy.Hosting.Self
Nancy.Owin ()
Other:

Other Nancy packages and versions:
Environment (Operating system, version and so on):
.NET Framework version:
Additional information:

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NancyFx/Nancy/issues/2823, or mute the thread
https://github.com/notifications/unsubscribe-auth/AEEilpTynmfvlwWfLotvhqxpIN3ULhKvks5s2wd3gaJpZM4QfIjj
.

The problem is, if you have a post request that might take a while to handle .
What about other requests incoming in that time?
You can do whatever multithreading you want inside that request, but the way it looks, if Nancy doesn't do multithreading, then no matter what you do, the request will hang and block everything else until you return from that function, and you need to return if you want to send data back, right? Or is there a way to achieve this in another method?

Are you asking if Nancy supports multiple concurrent requests? Of course it does. The request handling is mostly up to the host though. Nancy just takes the requests passed to it and handles it.

Or are you asking if you can spawn threads in Nancy modules? Why would you do that?


Upgrade com.puppycrawl.tools:checkstyle 10.6.0 -> 10.7.0

This PR contains the following updates:

Package
Type
Update
Change

com.puppycrawl.tools:checkstyle (source)
build
minor
10.6.0 -> 10.7.0

[ ] If you want to rebase/retry this PR, check this box

Suggested commit message:
Upgrade Checkstyle 10.6.0 -> 10.7.0

See:
- https://checkstyle.sourceforge.io/releasenotes.html
- https://github.com/checkstyle/checkstyle/releases/tag/checkstyle-10.7.0
- https://github.com/checkstyle/checkstyle/compare/checkstyle-10.6.0...checkstyle-10.7.0


Replace keystone module for os_* modules

roles/keystone/library/keystone is a custom ansible module used for various keystone tasks.
Siunce ansible <IP_ADDRESS> most of the tasks covered by keystone.py have been implemented in os_* modules shipped by ansible removing the need of keystone.py module.
The os_* modules requires shade library to be present while keystone.py requires opensatack-python-clients to be present.
The following tasks will be replaced as follows

keystone.py
ansible_module
added in

ensure_service
os_keystone_service
2.2

ensure_project
os_project
2.0

ensure_role
os_keystone_role
2.1

ensure_user
os_user
2.0

ensure_user_role
os_user_role
2.1

ensure_endpoint
os_keystone_endpoint
PR-20314

Those modules require shade wich is not yet available in ClearLinux


SPDX should allow custom license text

In melange one should be able to specify filepaths to custom license information.
spdx should generate reference to it and include the full license text.
This is to address license terms of "proprietary" stuff and custom license terms which are not yet (if ever will be) in spdx license list.

Reading https://spdx.github.io/spdx-spec/v2.3/other-licensing-information-detected/#102-extracted-text-field
it feels like maybe we want to add extracted license texts into our spdx documents, or possibly just create refs to external texts that we self manage.

  "hasExtractedLicensingInfos" : [ {
    "licenseId" : "LicenseRef-wolfi-dev-spdx-ubuntu-font",
    "extractedText" : "foobaz bar"
  }],
  "licenseDeclared": "LicenseRef-wolfi-dev-spdx-ubuntu-font",

is valid spdx. And I think we can generate that from a list of license files.

Maybe we can do something like this:
Given yaml that has
copyright:
    license-text:
        - LICENSE.txt
        - LICENSE.lib

Generate
"licenseDeclared": "Licenseref-1-font-ubuntu-0.869-r0 AND Licenseref-2-font-ubuntu-0.869-r0",

  "hasExtractedLicensingInfos" : [ {
    "licenseId" : "LicenseRef-1-font-ubuntu-0.869-r0",
    "extractedText" : "contents of LICENSE.txt"
  }, {
    "licenseId" : "LicenseRef-2-font-ubuntu-0.869-r0",
    "extractedText" : "contents of LICENSE.lib"
  }],

released https://github.com/chainguard-dev/melange/releases/tag/v0.8.0


[WIP] Storage-ready serialization

Includes #21; will be easier to review after that is merged.
This adds a new serialization cycle that (1) enables our objects to use the "path" identifiers described in #21; (2) allows our objects to ensure that a given path is only saved once, allowing (e.g.) protein files to be re-used instead of storage mutliple redundant copies.
The mechanism for this is a serialization instance method called to_storage_ready, which returns a dict mapping the object itself (since all our objects are hashable) to a namedtuple containing the bytes serialization, the path identifier, a cryptographic digest used as a label, the class of the object, and other possible metadata. If object A contains object B, and object B is to be stored in a separate file, then object A only stores the metadata about object B, and object B becomes another entry in the output dict. This dictionary can easily be ingested by the the storage system in #21, which just needs path and bytes.
The deserialization classmethod, from_storage_bytes, takes the bytes outputted by to_storage_ready as well as a function that can take a "path" identifier and return a the byte representation (in general, this function will be results_client.load_bytes for some instance results_client; passing this will be done behind the scenes).
Objects that have widely interoperable serialization formats (e.g., as SDF) are stored in those formats. Other objects (e.g., things specific to gufe) are stored as JSON objects.
In this PR, I'll add the core machinery, as well as support for several of our setup objects:

[x] SmallMoleculeComponent
[x] SolventComponent
[x] ChemicalSystem
[ ] Protocol
[ ] Transformation

So far, I'm excluding ProteinComponent because I'm not yet sure how we're serializing those.

Hello @dwhswenson! Thanks for opening this PR. We checked the lines you've touched for PEP 8 issues, and found:

In the file gufe/smallmoleculecomponent.py:

Line 17:1: E402 module level import not at top of file

In the file gufe/storage/utils.py:

Line 5:1: E302 expected 2 blank lines, found 1
Line 84:1: W391 blank line at end of file

Sorry that this is going to be a long comment, but I spent a lot of time trying to figure out ways to work around the problems I see, so this is kind of like a move decision tree where I have to tell you why every position I ended up in evaluated poorly in my view.
Original goal was to provide possible remediation strategies -- I'm too tired for that (beyond the approach that was already underway)
tl;dr
Moving these functions completely into storage will do the following:

Break discoverability.
Break duck typing.
Require registering all classes, which adds friction for external contributions
Make it harder to provide alternate storage systems
Break inheritance (unless you do a custom implementation of inheritance, and even that is going to have surprising behavior for users)

Options that I think might better meet what you want:

Drop to_dict and from_dict in favor of the more generalized serialization methods I'm writing here (inline the dict code into these methods). This is the direction I was trying to move here.
Maybe do a shallow to_dict/from_dict, and also require all objects carry a little extra information (specifically: label template and default values of to_dict). Then you can have storage be a consumer of these objects. (This more like what I've done in previous serialization systems, but I don't have the space to explore this now.)
It might be possible to use a nested GufeObject.Serialization class, which contains the extra info/methods. It could be used by, e.g., system.Serialization().get_storables(system) or ChemicalSystem.Serialization().from_storage(serialized_bytes, load_func). (Just thought of this; not entirely sure that it would work; no space to explore.)

1. Background: Goals and requirements for storage
1.1 We want to avoid duplication
Some of our objects may be common across many edges in a simulation campaign (e.g., protein representation, protocol settings, etc.) We don't want to save a new copy of these every time we save a ProtocolDagResult, so we need a mechanism for storing/loading by reference to a serialized object.
1.2 We might not want all objects to be stored by reference
There may be a point where we don't want separate data chunks (files) for each object. This is particularly true in settings: we might prefer to serialize the entirety of the protocol settings as a single data chunk. The reason is that, although this contains many small, reusable components, we might not particularly care that we're reusing a particular state point. In particular, one issue is that this result in the creation of many files in filesystem-based storage. We need a way to specify which objects should be saved by reference.
A reasonable alternative here is to say it's "turtles all the way down," and save everything by reference if we can (with the downside that it creates a lot of files).
1.3 Objects to store by reference might be in containers
Consider ChemicalSystem. We want to store its components by reference -- but these are the values in a dict. So we need to be able to identify objects in containers that should be saved by reference.
1.4 We use labels to store our objects
1.4.1 Data chunks are labelled
Since we are storing objects be reference, we need some way to refer to them. In keeping with the path-like labels we're using for result data, we use a path-like labels for our objects. We need a way to get the label for a given instance. We take the setup/ base "namespace" for this. For convenience, we further divide this namespace into other subpaces. Then each object is placed somewhere within this namespace with a file-like name. For example, a small molecule might have the label:
"setup/components/{object_identifier}.sdf"

1.4.2 A stable label
We want the object identifier to be stable across sessions, object creations, versions of our code.
Things that didn't work:

Python's hash: Not stable across sessions (salted with session identifier because of security concerns)
UUID: Not stable across creations. That is, if you create the same object twice, it will have a different UUID.
md5 (or other hash) of the JSON string for the to_dict() method: Not stable across minor versions of gufe. (Later minor versions may add new fields.)

The solution we've proposed is a cryptographic hash based on md5 of the non-default values in the to_dict of an object. While more complicated to say, it's basically a version of the last option that can preserve its value across changes in minor version.
This means that we need to determine which attributes are defaults and which are not. The approach I've take here is to ask the writer of the class to create a dict that maps the keys from to_dict to their default values, if such defaults exist. In many cases, this is trivial: see the defaults_from_init method for the convenience that works in many cases. However, there are plenty of cases where we cannot infer the defaults:
class Foo:
    def __init__(self, dict_attr=None):
        if dict_attr is None:
            dict_attr = {}
        self.dict_attr = dict_attr

A placeholder value can also correspond to more complicated, non-truthy attributes as well, so we can't just use truthiness here -- indeed, if the default is None and the user gave 0, these might have totally different meanings, so we can't make assumptions.
The information about what attributes are default is class-specific. Therefore it is an input from class writers to the storage system.
2. Why I don't like the suggested approach
2.1 This breaks discoverability
By keeping everything needed to implement a class in one place, we say to potential contributors "you can use this as a template for what you should do." The audience I'm thinking about here is a contributor who isn't a Python expert, and who just wants to toss in a quick addition. How does this contributor learn what needs to be done to add a new class?
Separating the serialization from the class immediately loses some discoverability. If I ask, "How do I serialize/deserialize this class?" I usually use something like tab completion to browse the class's methods. This approach hides that away. Even IPython's ?? won't show it.
There are two choices of where to put the serialization functions: in the same module as the class, or somewhere under storage. If you put it in the same module as the class, then I'm not sure what the point of separating them is, and I still don't like that the class isn't self-contained. If you put them in storage, then you really break discoverability. It means that a contributor who just wants to make a narrow contribution would need a wide knowledge of our code base. This seems like it would make it harder to onboard contributors, and be a likely source of confusion.
2.2 You can't use duck typing
The approaches I'm suggesting allow a contributor to add a new method that "just works" via duck typing. On the other hand, moving everything to storage requires a registration step, which makes it so that duck typing is not sufficient. New objects are not drop-in replacements for old ones, even if they implement the same API. I think a design that allows duck typing would be preferred to one that does not. Some of the issues below are specific examples where duck typing would solve the problem.
2.3 Registering classes creates an unnecessary extra step
For users to use classes from an external package, they'll need to register the class. For some external package, somewhere there needs to be code equivalent to this:
from externalpackage import Thing, serialize_thing, deserialize_thing, THING_DEFAULTS
import gufe.storage
gufe.storage.register(
    serializers={Thing: serialize_thing},
    deserializers={Thing: deserialize_thing},
    template_labels={Thing: "things/{object_id}"},
    defaults={Thing: THING_DEFAULTS},
)

Let's hope they don't make their users do that! But we have to tell the devs "in addition to writing your code, you need to write serialization registration code."
If the developers of externalpackage are at least somewhat kind, they'll put that in a register function. Still, it's an extra step for a user on import:
import externalpackage
externalpackage.register()

If they want to get rid of this, those devs might stick that in as a side effect during import. Side effects during import are generally considered a bad practice in Python, and I don't think we should encourage it. In this particular case, it isn't too terrible (since the side effect should be idempotent), but it is still a messy step that adds friction for an external contributor.
2.4 Registering classes makes it harder to use an alternate storage system
The additional information we provide is exactly the information that would be needed by an alternative storage system (where "storage" here means a replacement for our entire stack, from ResultsClient down). Note that the code above was for gufe.storage. If the developers of externalpackage also wanted to support alt_storage, they need to duplicate the registration code above to register with alt_storage. So alt_storage is no longer a drop-in replacement for our storage.
2.5 You can't use inheritance
Because subclasses might have different fields that need to be serialized, the lookup needs to be by class. So making a subclass (even one that doesn't change serialization) would require a separate registration.
To an extent, you can fake inheritance by manually traversing the MRO and trying a lookup for each class, taking the first one that succeeds. (This might not work for classes whose metaclass has a custom issubclass, but that's an edge case I'm willing to overlook.) This fake inheritance seems potentially confusing to me: how will a user know where to look when they encounter a bug in their serialization? When the relevant method is an instance method, I can easily find where it is defined.
2.6 Dropping consistent labels breaks potential use cases
I think there's some confusion about _storage_path. First, that's a class variable because it is intended to be something that could (in unusual circumstances) by modified without getting into the core code. It could have just as easily been hard-coded in the to_storage_ready method. Second, I think there's a misunderstanding: this is not a filename. Maybe it would be have better to call it _label_template: it is the template for the label that the storage uses to label its data.
Having that be consistent between storage systems makes sense. As I've said (over and over), that labelling schema is essentially part of the gufe API. How a storage system addresses these internally is not my problem, but since we deliver bytes that include these labels, those bytes shouldn't be dependent on the storage system used.
Weird use case: with consistent labels, you mix different storage systems as fallbacks. We could make fun_storage, that uses our ResultsClient as a first place to search, and then falls back on alt_storage if it couldn't load something.
3. What are the other options
3.1 Drop to_dict/from_dict
Currently, we could fold to_dict and from_dict into the other serialization methods. This is what I was generally moving toward. Right now, I think they're only used within these serialization methods. I think there's room for a cleaner API, but this might solve some problems.
3.x Other options
Other options exist. I want to complete these comment today for further discussion, and can't write a more exhaustive summary.

I'll do this as a PR against this branch. This will use dask's tokenization approach that we are using in #36.

Quick comment on this point, @dotsdl -- Our goal was to have tokens that are persistent across (minor) versions. See discussion in 1.4.2 of my long comment. I'm not sure that dask's tools can do that.

@dwhswenson the approach I'm pursuing (if it works out) should avoid the need for tokenization to be stable over time, so we can relax that requirement. We'd only use tokenization to avoid duplication in the storage system where possible, but it is only an optimization, not a guarantee. We won't rely on hashing/tokenization when retrieving anything; we could use UUIDs instead, but we would lose out on that storage space optimization.
I'll be pulling this thread for most of the rest of the day today. Wish me luck :sweat_smile:; will report back on where this landed, including downsides to the approach, in a PR.

We have data labeled as:
transformations/{TRANSFORMATION_HASH}/{RUN_REPLICA}/{EXTENSION}/{UNIT}/file.data?
If I can't get from Transformation to {TRANSFORMATION_HASH}, then how does Results know how to find all of the results for a transformation? Does it need to recreate every Transformation in the database, then do an (I think) O(N^2) operation to create equivalence classes by equality? After that it could loop through those to collect all the results.
Having a stable hash makes that an O(1) process.

A lot of the complexity here seems to come from wanting to avoid duplication in the storage medium, but this seems like a problem for the storage system to solve, not something that individual components need to solve.  All that should be required of objects that want to be stored is for them to define to_dict and from_dict.
If to_storage_ready was documented I might understand the problem better?

Avoiding store duplication IMHO feels like premature optimization at this point, I think starting out with that goal was a good one but it has started to reveal a lot of complexity. If people want to save money on storage they can just use a zfs deduplication setup and instead of spending money on storage they can spend money on ECC ram.

Closing this. We're relaxed the following requirements I was operating under, which made some of the additional complexity necessary:

not adding a base class
storing molecules as SDF
not changing the existing to_dict/from_dict methods


Modify a component new blazor

Problem encountered on https://dotnet.microsoft.com/en-us/learn/aspnet/blazor-tutorial/modify
Operating System: macos
Provide details about the problem you're experiencing. Include your operating system version, exact error message, code sample, and anything else that is relevant.
An unhandled exception occurred while processing the request.
InvalidOperationException: Object of type 'BlazorApp.Components.Pages.Counter' does not have a property matching the name 'IncrementAmount'.
Microsoft.AspNetCore.Components.Reflection.ComponentProperties.ThrowForUnknownIncomingParameterName(Type targetType, string parameterName)

Can you share the contents of your Counter.razor page?
Does it look like the code that is shown on the page? Because the new property is being added on this step.


Release v0.24.1

v0.24.1 May. 17, 2021
Enhancements

Integrated ARIMARegressor into AutoML #2009
Updated HighlyNullDataCheck to also perform a null row check #2222
Set max_depth to 1 in calls to featuretools dfs #2231

Fixes

Removed data splitter sampler calls during training #2253
Set minimum required version for for pyzmq, colorama, and docutils #2254
Changed BaseSampler to return None instead of y #2272

Changes

Updated pipeline repr() and generate_pipeline_code to return pipeline instances without generating custom pipeline class #2227

Documentation Changes

Capped Sphinx version under 4.0.0 #2244

Testing Changes

Change number of cores for pytest from 4 to 2 #2266
Add minimum dependency checker to generate minimum requirement files #2267

Perf test results are here.


[ENG-6505][eas-cli] Fix automatic configuration if eas.json does not exist.

Checklist

[ ] I've added an entry to CHANGELOG.md if necessary. You can comment this pull request with /changelog-entry [breaking-change|new-feature|bug-fix|chore] [message] and CHANGELOG.md will be updated automatically.

Why
Code that is handling deprecated eas.json does not handle a case where eas.json does not exists and fails before automatic configuration has a chance to trigger.
How
Test Plan
eas build on a project without eas.json

/changelog-entry bug-fix Fix automatic eas.json generation when running eas build.


Unable to show devices on SmartThings Integration

I tried many times to integrate SmartThings under hassio (hassos), but under logs i found some errors like this... some ideas?
Error setting up entry Domov for smartthings
ValueError: 'OCF' is not a valid DeviceType
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 192, in async_setup
    hass, self
  File "/usr/src/homeassistant/homeassistant/components/smartthings/__init__.py", line 116, in async_setup_entry
    devices = await api.devices(location_ids=[installed_app.location_id])
  File "/usr/local/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 77, in devices
    return [DeviceEntity(self._service, entity) for entity in resp]
  File "/usr/local/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 77, in <listcomp>
    return [DeviceEntity(self._service, entity) for entity in resp]
  File "/usr/local/lib/python3.7/site-packages/pysmartthings/device.py", line 651, in __init__
    self.apply_data(data)
  File "/usr/local/lib/python3.7/site-packages/pysmartthings/device.py", line 94, in apply_data
    self._type = DeviceType(data['type'])
  File "/usr/local/lib/python3.7/enum.py", line 310, in __call__
    return cls.__new__(cls, value)
  File "/usr/local/lib/python3.7/enum.py", line 564, in __new__
    raise exc
  File "/usr/local/lib/python3.7/enum.py", line 548, in __new__
    result = cls._missing_(value)
  File "/usr/local/lib/python3.7/enum.py", line 577, in _missing_
    raise ValueError("%r is not a valid %s" % (value, cls.__name__))
ValueError: 'OCF' is not a valid DeviceType

I'd had a similar issue after adding a new Samsung TV to Smartthings when I updated HA (I'm running on Docker) I got the same issue. I just logged into the Samsung Graph API and remove the TV and it worked ok again. I suspect it was the DeviceType (which was set to undefined on the API) I haven't tried readding as yet

I'd had a similar issue after adding a new Samsung TV to Smartthings when I updated HA (I'm running on Docker) I got the same issue. I just logged into the Samsung Graph API and remove the TV and it worked ok again. I suspect it was the DeviceType (which was set to undefined on the API) I haven't tried readding as yet

hmm. sounds familiar... i added new SamsungTV to smart things app too.

It looks like the API changed in how it represents OCF devices.  I'll update the library and submit a PR today to address this.  Sorry for the delay folks!

Hit this today with DeviceType “HUB”. Updated to 0.104.3 and it went away. Thanks @andrewsayre

Same here, sorted the problem quickly.

I ma having the same issue and I am running 0.105.3:
2020-02-12 11:42:39 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Home for smartthings
ValueError: 'HUB' is not a valid DeviceType
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/config_entries.py", line 190, in async_setup
hass, self
File "/srv/homeassistant/lib/python3.7/site-packages/homeassistant/components/smartthings/init.py", line 116, in async_setup_entry
devices = await api.devices(location_ids=[installed_app.location_id])
File "/srv/homeassistant/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 77, in devices
return [DeviceEntity(self._service, entity) for entity in resp]
File "/srv/homeassistant/lib/python3.7/site-packages/pysmartthings/smartthings.py", line 77, in
return [DeviceEntity(self._service, entity) for entity in resp]
File "/srv/homeassistant/lib/python3.7/site-packages/pysmartthings/device.py", line 651, in init
self.apply_data(data)
File "/srv/homeassistant/lib/python3.7/site-packages/pysmartthings/device.py", line 94, in apply_data
self._type = DeviceType(data['type'])
File "/usr/lib/python3.7/enum.py", line 310, in call
return cls.new(cls, value)
File "/usr/lib/python3.7/enum.py", line 564, in new
raise exc
File "/usr/lib/python3.7/enum.py", line 548, in new
result = cls.missing(value)
File "/usr/lib/python3.7/enum.py", line 577, in missing
raise ValueError("%r is not a valid %s" % (value, cls.name))
ValueError: 'HUB' is not a valid DeviceType

^^^
bump

it would appear that the normal command was not not updating home assistant:
pip3 install --upgrade homeassistant
I have it all working now but had to issue the following:
sudo hassbian-config upgrade homeassistant


Java prime-factors problems with tests

Aside from test case 0, I get all the tests failing with messages like:
"java.lang.AssertionError: expected: java.util.Arrays$ArrayList<[2, 3]> but was: java.util.ArrayList<[2, 3]>"

The lists match and the values have the same orders. Initially I was using a LinkedList and thought that was the problem, but changing to ArrayList didn't solve the problem.
Oddly enough, test 0 pass (empty list), probably because the initalizer is
Arrays.asList(new Integer[0]);

instead of
Arrays.asList(new Integer[]{})

as the others are.
I'm using IntellijIdea 14.1 , running as JUnit tests. Attached the code for my solver:
public class PrimeFactors {
    public static List<Long> getForNumber(double input) {
        List<Long> result = new ArrayList<>();
        long number = (long) input;

        //Primality checks only up to SQRT(num)
        for (long i = 2; i * i <= input; i++) {
            //Eratosthenes-like approach at removing prime multiples
            while (number > 1 && number % i == 0) {
                number /= i;
                result.add(i);
            }
            //Premature exit if we are done
            if (number <= 1) {
                break;
            }
        }
        //Leftovers are prime numbers too
        if (number > 1) {
            result.add(number);
        }
        return result;
    }
}

That's weird (and frustrating).
I think the tests are expecting the type to be Integer not Long Is there a way to write the tests so that it would accept either? It seems to me that the underlying type is not important in this case so long as the code produces the correct results.

The quickest fix I can think of is
assertEquals(expectedOutput -  PrimeFactors.getForNumber(input), 0);

Otherwise we could explicit add casts to long for both sides of the assert.
A bit hacky, but working. Maybe JUnit has something better to offer, though I'm not aware of it.

Also, the last test passes in a Long. While its factors are all within the Integer range, it's probably best to just convert the whole test to use all longs?

Yeah, let's just use longs everywhere.

Otherwise we could explicit add casts to long for both sides of the assert.

That might not be a bad idea, actually. The downside is it might look like we think it's a good practice, whereas it would probably be best to avoid in the real world.

Given that the last test requires a Long input, I also prefer going with longs everywhere.


Make commons-compress an optional dependency

I'm experiencing difficulties with ProGuard and commons-compress. This has led me to consider the possibility of excluding the commons-compress dependency, allowing the application to function smoothly unless a zipped model that requires unzipping is explicitly specified. This approach would enable me to circumvent ProGuard issues associated with commons-compress while reducing unnecessary dependencies.

Can you exclude commons-compress from your project if you are not using any .tar file?
This should be pretty easy from gradle:
    implementation platform("ai.djl:bom:0.26.0")
    implementation("ai.djl:api") {
          exclude group: "org.apache.commons", module: "commons-compress"
    }

@frankfliu Do you have an estimate when the next release including this fix will be available?
I see you seem to have monthly releases.

We just released 0.26.0, 0.27.0 will be in March, see: https://docs.djl.ai/index.html#release-notes
For the meantime, you can use our nightly snapshot release: https://docs.djl.ai/docs/get.html#nightly-snapshots

@frankfliu I've tested the latest snapshot version and found that it functions smoothly with ProGuard. There seems to be something off with commons-compress that hinders optimization. I'm glad I don't need that library as long as I'm not using TAR files. Thank you for quickly fixing this!
Considering March is a considerable time away, I was wondering if you could consider releasing a v0.26.1 fix version. I'm a bit hesitant to deploy a snapshot version in a production environment, as it might introduce breaking changes at any moment.

@zachgk Thank you for showing me a workaround. I guess I need to try that then.
I still must say I’m a bit surprised that you have so long and fixes release schedules and that it’s such an effort to do a release. Sounds to me like something is off.
Maybe you should consider reworking your CI pipeline for more automation. I maintain two libraries with Maven Central artifacts and releasing a new one is just a matter creating a GitHub release (which builds everything using GitHub actions and uploads to Maven Central) and confirming in the Sonatype Nexus that the artifacts should be released to Maven Central.

@StefanOltmann
Releasing DJL is non-trivial work. We have 11 engines and need to test on to following platforms:

macOS
macOS M1
ubuntu

x86_64
aarch64
cuda 112, cuda 113, cuda 118
cuda 12.1

amazlinux:2 x86_64 and aarch64
centos 7
Widnows server 2019+

For each release, we publish over 70+ artifacts. And a we more than 6 downstream github repo (djl-serving, djl-demo, springboot-start, docs, jupyter notebooks etc) to be updated to new version.

The artifact count shouldn’t matter with the automation, but I see your point with the downstream repos.


Support lb:// url in FeingClient annotation

Current behaviour
In a project where is used spring cloud gateway, feign client (with target put in configuration), we ended up having feign clients declared as follows
@FeignClient(name="${service.name:any-name}", url="${service.direct-url:}")

With configuration as follows
service:
   name: service-id
   # url: http://direct-url (only used when service discovery is not available)

This way we are able, by configuration, to either run in service discovery/load balancing  mode and direct mode.
Proposal
I kind of like the way spring cloud gateway handle that by allowing lb:// protocol. Allowing this protocol in url attribute would allow the  following implementation
@FeignClient(name="technical-name", url="${service.uri}")

with following configuration
service:
   uri: "lb://service-id" # or "http//direct-url"

this way it will let common behaviour, allowing to have a common behaviour in url specification.
I spotted the code responsible for feign client building and think that it would be easily doable by

only appending  http:// if url attribute value does not match the [a-z]+://.* pattern
remove load balancing client if and only if url attribute does not starts with lb://

I am eager to implement it if you allow me to do so. I am also pleased to discuss about it and about any alternative I missed.
Thank you

@OlgaMaciaszek what the point on this issue? Can I make a PR?

PR submitted

@antechrestos  Actually, I am not sure this is a change we want to make. We are going to see if there's interest in the community and, if yes, then, we will discuss it in the team.

@OlgaMaciaszek Agreed. Just for clarification, is there any documentation describing "the waiting for votes" adhesion and what is the comunity? Is there a number of :+1:  expected on the issue? Or is it based on an adhesion of an inner community (spring developers for instance)?
Thank you

We check how many users request a feature. If we see considerable interest, then we discuss the possible issues and implications within the Spring Cloud team, taking into account the discussion in the issue and other considerations related to maintenance, planned future project development and possible issues.

@OlgaMaciaszek  ok that's more clear

To my mind, I think the real question is if there is value to apply the concept of lb:// url used in spring cloud gateway in other spring projects or if it needs to be restricted to it.
As an alternative, it may be useful to let the user decide whether (s)he wants to skip load balancing/discovery behaviour. Because currently, the url use hides the fact that we force the use of delegated client... What do you think of it?

@OlgaMaciaszek do you think I should open a new issue proposing to handle load-balancing/discovery in a dedicated properties rather than hidden behind url attribute?


Update table azure_api_management

References
https://docs.microsoft.com/en-us/rest/api/apimanagement/2021-01-01-preview/api-management-service/list
Note: This detail is required for NIST SP 800 53 Rev 5 controls.

The azure_api_management table was already present. My PR will update the table based on the properties that we need


Kubernetes core concepts for Azure Kubernetes Service (AKS)

[Enter feedback here]

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: a445f8cc-6581-ad43-1dba-719a94eda81a
Version Independent ID: 31280746-d8de-91b4-d430-91693c2e869d
Content: Concepts - Kubernetes basics for Azure Kubernetes Services (AKS) - Azure Kubernetes Service
Content Source: articles/aks/concepts-clusters-workloads.md
Service: azure-kubernetes-service
GitHub Login: @MGoedtel
Microsoft Alias: magoedte

@kamaljitsingh76
Could you please elaborate the query for better understanding. So, we can assist you further.

Closing this issue as a result of insufficient details helping to understand what after-action is required. You can always reopen this issue and provide us with information that can help us understand what your concern is with the article. Thanks.  #please-close


Add writing setting

What I am changing
In our usage of tipg (lambdas and Aurora on AWS) we're connecting to a database endpoint which is read-only, so I've added a setting for disabling the writing of the functions and the custom SQL to the database.
I haven't added docs yet, but at this stage I wanted to check in if you're interested in merging this and if so if the setting is in the right category or if there would be a better category (e.g. PostgresSettings)

tipg should be read-only. We don't really write to the database but use the pg_temp schema to register functions (user custom functions and tipg internal functions) dynamically. Those get deleted when the connection is deleted.

tipg should be read-only. We don't really write to the database but use the pg_temp schema to register functions (user custom functions and tipg internal functions) dynamically. Those get deleted when the connection is deleted.

There's another pull request I have waiting to allow the use of schemas other than pg_temp (because Aurora doesn't allow the use of pg_temp) to help with that issue, but the user should indeed create the functions by hand when using this setting and this should be specified in the docs as well.
But I do realize that this is relatively niche issue, there won't be many users who have a read-only db connection or have a db user with stripped down rights but sadly we are one of them. Having this merged is mostly nice so we don't have to maintain our own fork even if it does mean slightly more work in regards to keeping the SQL functions up to date :slightly_smiling_face:

because Aurora doesn't allow the use of pg_temp

Ah I see 😭

I do realize that this is relatively niche issue, there won't be many users who have a read-only db connection or have a db user with stripped down rights but sadly we are one of them. Having this merged helps us insofar that we don't have to maintain our own fork. Then we only have the work of manually keeping the SQL functions up to date 🙂

TBH you don't have to maintain your own fork but create a customized application using something like this 👇
import datetime
from typing import Any, List, Optional

import asyncpg
import orjson
from fastapi import FastAPI

from tipg.collections import Catalog, Collection
from tipg.settings import TableSettings, PostgresSettings

async def get_collection_index(  # noqa: C901
    db_pool: asyncpg.BuildPgPool,
    schemas: Optional[List[str]] = None,
    tables: Optional[List[str]] = None,
    exclude_tables: Optional[List[str]] = None,
    exclude_table_schemas: Optional[List[str]] = None,
    functions: Optional[List[str]] = None,
    exclude_functions: Optional[List[str]] = None,
    exclude_function_schemas: Optional[List[str]] = None,
    spatial: bool = True,
) -> Catalog:
    """Fetch Table and Functions index."""
    schemas = schemas or ["public"]

    # Replace pg_temp by public
    query = """
        SELECT public.tipg_catalog(
            :schemas,
            :tables,
            :exclude_tables,
            :exclude_table_schemas,
            :functions,
            :exclude_functions,
            :exclude_function_schemas,
            :spatial
        );
    """  # noqa: W605

    async with db_pool.acquire() as conn:
        rows = await conn.fetch_b(
            query,
            schemas=schemas,
            tables=tables,
            exclude_tables=exclude_tables,
            exclude_table_schemas=exclude_table_schemas,
            functions=functions,
            exclude_functions=exclude_functions,
            exclude_function_schemas=exclude_function_schemas,
            spatial=spatial,
        )

        catalog: Dict[str, Collection] = {}
        table_settings = TableSettings()
        table_confs = table_settings.table_config
        fallback_key_names = table_settings.fallback_key_names

        for row in rows:
            table = row[0]
            table_id = table["schema"] + "." + table["name"]
            confid = table["schema"] + "_" + table["name"]

            # Replace pg_temp by public
            if table_id == "public.tipg_catalog":
                continue

            table_conf = table_confs.get(confid, {})

            # Make sure that any properties set in conf exist in table
            properties = sorted(table.get("properties", []), key=lambda d: d["name"])
            properties_setting = table_conf.get("properties", [])
            if properties_setting:
                properties = [p for p in properties if p["name"] in properties_setting]

            # ID Column
            id_column = table_conf.get("pk") or table.get("pk")
            if not id_column and fallback_key_names:
                for p in properties:
                    if p["name"] in fallback_key_names:
                        id_column = p["name"]
                        break

            datetime_column = None
            geometry_column = None

            for c in properties:
                if c.get("type") in ("timestamp", "timestamptz"):
                    if (
                        table_conf.get("datetimecol") == c["name"]
                        or datetime_column is None
                    ):
                        datetime_column = c

                if c.get("type") in ("geometry", "geography"):
                    if (
                        table_conf.get("geomcol") == c["name"]
                        or geometry_column is None
                    ):
                        geometry_column = c

            catalog[table_id] = Collection(
                type=table["entity"],
                id=table_id,
                table=table["name"],
                schema=table["schema"],
                description=table.get("description", None),
                id_column=id_column,
                properties=properties,
                datetime_column=datetime_column,
                geometry_column=geometry_column,
                parameters=table.get("parameters", []),
            )

        return Catalog(collections=catalog, last_updated=datetime.datetime.now())

async def register_collection_catalog(app: FastAPI, **kwargs: Any) -> None:
    """Register Table catalog."""
    app.state.collection_catalog = await get_collection_index(app.state.pool, **kwargs)

class connection_factory:
    """Connection creation."""

    schemas: List[str]
    user_sql_files: List[pathlib.Path]

    def __init__(
        self,
        schemas: Optional[List[str]] = None,
    ) -> None:
        """Init."""
        self.schemas = schemas or []

    async def __call__(self, conn: asyncpg.Connection):
        """Create connection."""
        await conn.set_type_codec(
            "json", encoder=orjson.dumps, decoder=orjson.loads, schema="pg_catalog"
        )
        await conn.set_type_codec(
            "jsonb", encoder=orjson.dumps, decoder=orjson.loads, schema="pg_catalog"
        )

        # Note: we add `pg_temp as the first element of the schemas list to make sure
        # we register the custom functions and `dbcatalog` in it.
        schemas = ",".join(["pg_temp", *self.schemas])
        logger.debug(f"Looking for Tables and Functions in {schemas} schemas")

        await conn.execute(
            f"""
            SELECT set_config(
                'search_path',
                '{schemas},' || current_setting('search_path', false),
                false
                );
            """
        )

async def connect_to_db(
    app: FastAPI,
    settings: Optional[PostgresSettings] = None,
    schemas: Optional[List[str]] = None,
    **kwargs,
) -> None:
    """Connect."""
    if not settings:
        settings = PostgresSettings()

    con_init = connection_factory(schemas)

    app.state.pool = await asyncpg.create_pool_b(
        settings.database_url,
        min_size=settings.db_min_conn_size,
        max_size=settings.db_max_conn_size,
        max_queries=settings.db_max_queries,
        max_inactive_connection_lifetime=settings.db_max_inactive_conn_lifetime,
        init=con_init,
        **kwargs,
    )

async def close_db_connection(app: FastAPI) -> None:
    """Close connection."""
    await app.state.pool.close()

I understand this is not ideal

We did go down that route, but that also ended up with us basically maintaining our own fork. So these changes (if they're not merged) minimize the amount of work to keep our fork in sync with the main version due to sharing the same git history and us not touching any of the actual logic.

👍 let see what @bitner thinks

We talked internally with @bitner, right now we are trying to focus on making tipg working well with Postgres and Aurora is not really Postgres but Postgres compatible.
I think the best way forward is to create a kinda tipg backend extensions. I can start a PR to see how this will look.
Note: a backend should just provide a Catalog
class Catalog(TypedDict):
    """Collection Catalog."""

    collections: Dict[str, Collection]
    last_updated: datetime.datetime

The Collection should be a child object from tipg.collections.Collection


[Java] Build memory-netty as a JPMS module

Describe the enhancement requested
This depends on #38998.
Build memory-netty as a JPMS module.
This change requires patching Netty's public module since we add classes to exposed packages and call package-private functions.
This requires users to add a patch-module command to their java invocation. It is somewhat inconvenient because it needs to reference one of our delivered JAR filenames (which vary by location and version)
Component(s)
Java

Hi @jduo, maybe I’m mistaken, but is this the only missing item for https://github.com/apache/arrow/issues/38915?
Thanks!

Thanks for the super quick answer. The reason I’m asking this is because it seems that the snowflake driver needs this dependency upgraded to support Java > 16. If this is not going to happen then I guess that the snowflake team is stuck?

add classes to exposed packages and call package-private functions.

What classes are added and which package-private functions are called? If this functionality is required, should there be a conversation with netty wrt getting those packages opened?

All of the content in arrow-netty-buffer-patch either depends on Netty internals or depends on classes that is depend on Netty internals.
I think the most prevalent was Buffer.
This work seems to have been used as a way of optimizing Flight/gRPC integration.

What are the function in question? Which part of Arrow is affected? I'm interested in this issue


nvidia-settings no longer required by any nvidia packgage

Just noticed that nvidia-settings package is no longer required by any othe nvidia subpackgage. Not sure if this was by intention. It happened with PR #28.

Yes it is intentional. Any information you can get with nvidia-settings -q can be derived from other tools.
In the specific case of a normal laptop (i.e. an Optimus laptop) with just nvidia-video-G06 and nothing else it creates more confusion than benefits, as user thinks they need to use that to configure X.
With Wayland, has almost no use anymore, except for querying some information. I would live it as optional.

Thanks. That's great news. Nothing to change here! :-)


Could not create config file

I have captured and trained the module using terminal on my mac, but when adding to the config file and adding classes to each other module, I continue to get errors in the config. Before adding the facial recognition module, the magicmirror would start up without a problem, now adding the single module creates a config problem. I am very new to this and just wondering if i am missing something simple, thanks!

Update, no longer getting config errors, instead getting this...

Seems like OpenCV can't find training.xml from the path you give in MagicMirror configs.js
trainingFile: "/jacoblevato/MagicMirror/modules/MMM-Facial-Recognition-OCV3/training.xml"
Should this be
trainingFile: "/Users/jacoblevato/MagicMirror/modules/MMM-Facial-Recognition-OCV3/training.xml"
Also it seems like the users field in you config is empty. When you train the face models with the python script it should give you the list of trained users that you should put in this field.


[Snyk] Security upgrade ruby from 2.6.5 to 2.7.6

Snyk has created this PR to fix 2 vulnerabilities in the dockerfile dependencies of this project.
Keeping your Docker base image up-to-date means you’ll benefit from security fixes in the latest version of your chosen image.
Snyk changed the following file(s):

openshift-configs/Dockerfile

We recommend upgrading to ruby:2.7.6, as this image has only 550 known vulnerabilities. To do this, merge this pull request, then verify your application still works as expected.
Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Out-of-bounds Write SNYK-DEBIAN10-LIBWEBP-5893093
  852

Out-of-bounds Write SNYK-DEBIAN10-LIBWEBP-5893093
  852

Out-of-bounds Write SNYK-DEBIAN10-LIBWEBP-5893093
  852

Out-of-bounds Write SNYK-DEBIAN10-LIBWEBP-5893093
  852

CVE-2023-44487 SNYK-DEBIAN10-NGHTTP2-5953390
  561

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: code/snyk check is complete. No issues have been found. (View Details)


how to convert chinese voice to the npy?

as I learnd from your project,
there is two type of files needed to train bark.
the npy and wav.
how to generate the npy files?
thanks alot

+1
I am trying to do the same for italian

as I learnd from your project, there is two type of files needed to train bark. the npy and wav. how to generate the npy files? thanks alot

Solved it, just read at the code, create_data.py actually call a function inside data.py, it load some ebooks in ram and process them generating .npy files.
I had to modify the code because i have some ebooks locally downloaded. this is my data.py for an italian dataset:
import random
import os

allowed_chars = ' abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZÙÚÀÁÓÒÉÈÌÍàáèéìíòóùú0123456789!@#$%^&*()-_+=\"\':;[]{}/<>,.`~\n\\'
dir = '/mnt/m/projects/ebook_txt_ita'

def load_book(dir, book):
    data = open(dir + '/' + book, 'r').read().encode().decode().replace('\n', ' ')
    return data

def filter_data(data, book):
    print('Filtering data on:', book)
    return ''.join([char for char in data if char in allowed_chars])

def load_books():
    text_data = []
    print(f'Loading books into ram')
    #for book in books:
    for book in os.listdir(dir):
        text_data.append(filter_data(str(load_book(dir, book)), book))
    print('Loaded books')
    return ' '.join(text_data)

def random_split_chunk(data, size=14):
    data = data.split(' ')
    index = random.randrange(0, len(data))
    return ' '.join(data[index:index+size])


Change map icons for sites with a buoy, and add HOBO icons for sites with a HOBO logger (icons attached)

icons.zip

Hi, i would like to work on this issue. Please guide me a bit more

The spotter icon should be blinking (see screen recording). The icons should have the white outline when selected.
https://user-images.githubusercontent.com/6283065/113486412-60b5c680-9467-11eb-8ee8-7e9e54d79043.mov

Hi @himanshu007-creator please have a try yes!
As a first step you can look at how we get the current pins for the map through findIntervalByLevel in website/src/helpers/bleachingAlertIntervals.ts


buildEnv.pl problems

Issue description
When i try to install binutils, this happens:
% nix-env -f "<nixpkgs>" -iA binutils --show-trace
installing ‘cctools-binutils-darwin’
building path(s) ‘/nix/store/r6dq1hy9a9kzmrz36cscyjy05frm36nw-user-environment’
Wide character in die at /nix/store/64jc9gd2rkbgdb4yjx3nrgc91bpjj5ky-buildenv.pl line 87.
error creating link ‘/nix/store/r6dq1hy9a9kzmrz36cscyjy05frm36nw-user-environment/share’: File exists at /nix/store/64jc9gd2rkbgdb4yjx3nrgc91bpjj5ky-buildenv.pl line 87.
builder for ‘/nix/store/hbwm86gy1911viszpab0d03n6zdpsy2m-user-environment.drv’ failed with exit code 17
error: build of ‘/nix/store/hbwm86gy1911viszpab0d03n6zdpsy2m-user-environment.drv’ failed

Steps to reproduce
nix-env -f "<nixpkgs>" -iA binutils
Technical details

System: OS X 10.10.5
Nix version: 1.11.2
Nixpkgs version: 16.09pre83147.df89584

The error is not during building binutils. It's during buildEnv: probably some bug in it triggered by the combination you've installed by nix-env. The bug is probably "portable". IIRC there were problems if there's a symlink-directory collision. Note that buildEnv is about to get rewritten soonish and you're affected by the copy living in a different repository.

Could you retest this now? https://github.com/NixOS/nixpkgs/pull/16828 was merged and might have solved the issue

I get the same result with nixpkgs-unstable channel.

@dredozubov could you please post the output of nix-env -q here (as @vcunat noted, the issue is most probably caused by your particular combination of installed packages).

Closing. This has been fixed for a while. Feel free to reopen it if it's still an issue. Thanks!


Add ftl::Fibtex

Here's the implementation of a mutex for fibers. Closes #64.

Overall, the code changes are great. Just small comments / documentation stuff for fixes. I really like the FTL_PAUSE additions to the rest of the code.

Hey, looks great so far! Fibtex is missing thread pinning support, which is unfortunately something that is needed for adoption.

@martty Thanks! How would you expect this to be implemented? Should the mutex always resume on the same thread, or should it just have an option with the lock functions to be able to do that (defaulted parameter).

I'm also going to add some RAII helpers for things std::lock_guard can't deal with.

@martty Thanks! How would you expect this to be implemented? Should the mutex always resume on the same thread, or should it just have an option with the lock functions to be able to do that (defaulted parameter).

Following the API of WaitForCounter, I think a defaulted ctor argument would be fine (default = unpinned).

Can you explain why you chose to implement three lock() functions?

At the bare minimum I wanted to implement a yielding lock and a spin lock. I split up the two spin lock functions because the behavior of .spin_lock(0) is ambiguous (could mean block immediately, could mean never block), and it eliminated extra checks inside the spin_lock function.

Having multiple lock*() functions makes it more difficult to use, since only lock() is integrated with STL (calling these by hand is sure way to foot shooting IMO :) ). What do you think of specifying the spinning mechanism in FibTex?

@martty This is why I am currently working on specific lock guards for use with these functions. I'll push them momentarily for comment.

I want to avoid making the mutex itself spin/yield specific, as you might want to have it spin in some situations but yield in others.

Sure, I meant specifying the spinning mechanism like a constructor parameter or a template parameter.

That's what I mean, sorry if I wasn't clear. I don't want to force a particular instance of a lock to be bound to a locking type because you may need to use the same lock in different ways depending on the situation. If fiber 0 is just updating a flag guarded by the lock, but fiber 1 is updating the whole data structure, Fiber 0 will want to do a blocking lock, and fiber 1 will want to do a spin lock.

I have implemented lock guards and I'll implement unqiue locks in a bit, but I need to go home, so I put these up for review in the mean time :smile:

Nice job!

Alright, I've added a basic Unique Lock impl. I have followed the standard (i.e. cppreference) to the letter with the exception of the swap function being a friend function to replace the the specialization of std::swap (which isn't possible). If you think we should deviate from the standard at any point lmk. I'll get on documenting and building the 2 other unique locks.

Can you make FibTex noncopyable and nonmovable?

@ChemiaAion I'll do a pass over all of the members and make them as restrictive as possible.

Still need to do a round of commenting and such, but the functionally should all be there at this point unless someone thinks of something else that should be there.

This PR should be ready to go in terms of functionality. Github is still a little hiccup-ey and we still have that random segfault bug in the tests. I might also add some tests to make sure it clears all the compilers.

I want to thank you so much for your work implementing this. I apologize for taking so long to get back to you with your latest changes.
I really like the main implementation. One comment is, perhaps we can get rid of the custom std guards etc, by putting all the logic in WrapperLock. For example: https://github.com/RichieSams/FiberTaskingLib/commit/84657942e831db86bf5ca7d70d04562b5e04be6d
I also added some super simple tests / example code in this one: https://github.com/RichieSams/FiberTaskingLib/commit/4dcf623a450575036508d7895a46e7885fc005f3
What are you thoughts? Feel free to cherry pick and/or edit.

I mean, it's ultimately up to you. I am just leaning very strongly towards the idea that you can just replace std with ftl and it will all work™. Having an identical interface is something that will make people better able to read the code, as well as write it as everyone has already figured out the std:: interface. Now they just use the same interface and it just works. You also don't have that layer of indirection which could lead to bugs due to differing scopes..

So some updates on the tests. There is some weird interactions between ftl::UniqueLock and ftl::lock. After the locks should have been fully released, there is still a used slot in the wait queue of the lock's AtomicCounter. Not sure what is causing it, but I've narrowed it down pretty well.

Good idea with the configurable NUM_WAITING_FIBER_SLOTS. Perhaps we could have two constructors for the atomic counter. The base constructor just uses the internal slots. The other constructor would take the size as an argument, and would dynamically allocate.

I was definitely thinking that, I'm trying to figure out what the best way to go about that is. That will be another PR, as this one is growing much too large and unwieldy as it is, it has tendrils basically everywhere.

As a slight tangent (I should add this to the style guide), I prefer rebasing / fastforwarding over merge commits as much as possible.

The biggest thing that seems to be blocking ftl::Fibtex from merging is this race condition that is described in #75.

I am going to close this to prevent it from cluttering things up because I no longer have the time to work on it, and I have basically abandoned c++ for my projects. Keep up the good work y'all.


Remove synchronization between GlobalSnapshotManager and ComposeScene.postponeInvalidation

The synchronization between GlobalSnapshotManager and ComposeScene.postponeInvalidation attempts to make sure that any modified state is applied (notifications sent) before the next frame is executed. However following discussions with Chuck Jazdzewski from Google, it appears that this is not achievable, and in fact should not be a goal. The only guarantee made by compose-runtime is that any changes in state will eventually be seen in compose.
Proposed Changes
Remove the synchronization.
Testing
Test: Manual test run
Issues Fixed
Fixes: https://github.com/JetBrains/compose-multiplatform/issues/1854

Just to make it more clear to me:
This PR is followup to https://github.com/JetBrains/compose-multiplatform-core/pull/543 ?
So #543 also fixes https://github.com/JetBrains/compose-multiplatform-core/pull/199 ?

@eymar

Just to make it more clear to me: This PR is followup to #543 ?

They're both related to the GlobalSnapshotManager, but it's a different issue here. The synchronization is just not needed.

So #543 also fixes #199 ?

If you mean whether the fix in #199 still holds and this PR doesn't break https://github.com/JetBrains/compose-multiplatform/issues/1392 again, I will check that.

@eymar https://github.com/JetBrains/compose-multiplatform/issues/1392 doesn't reproduce, and https://github.com/JetBrains/compose-multiplatform/issues/1396 doesn't reproduce either (probably fixed by https://github.com/JetBrains/compose-multiplatform-core/pull/532


OS-Climate - Establish the base software versions and tools for Stable instance cluster

As OS-Climate environment system administrator I need an environment where I can test changes to the configuration and new versions of the applications without interrupting the work of data scientists.
Proposal:
Provision new OpenShift Cluster for OS-Climate project that will become production environment while keeping existing cluster with scaled down capacity for testing of new frameworks, components and configuration changes.
Some of the proposed requirements:

Should include separate infrastructure nodes to host monitoring/alerting, authentication workload
OCP cluster needs to be provisioned in availability zones that provide GPU enabled EC2 instances (p3 instances)
Nodes hosting jupyter notebooks should have taint to prevent any other workload to be scheduled to these nodes. Notebooks that require GPU resources will need to have toleration for gpu-enabled taints

cc @MichaelTiemannOSC @caldeirav

what's the actual purpose here? from what I read we want a test env and declare the existing one stable?

@durandom We want to have test/staging environment and stable/production environment for data scientists. New cluster will become this stable/production and existing cluster will become dev/staging .We probably will need to be re-built dev cluster eventually to reflect any changes in configuration and processes that have been used for provisioning of production environment

@redmikhail I would actually recommend having two new clusters for DEV and PROD, and leave the existing MVP cluster running until we confirm that the new clusters work (starting with DEV), and retire it once confirmed. The reason is we want to have a clean transition to a new environment (with new address) including starting with clean storage volumes as well. This was discussed in our OS-C call yesterday and would make it easier to manage the change without impacting developers who are still on the existing cluster.

@caldeirav Sorry I was under impression that we want to re-use existing cluster as dev , will add it to the tasks

@redmikhail we would like to move forward with the creation of the production cluster now, before upgrading the Trino component on DEV (i.e. initially the two clusters will be in sync, then we will start working a validating changes in Dev before promoting to Prod). Do we have the required resources to create Prod? Let us know if any dependency please.

Calculate quota increase and look at needs .  Airbus will decide to move based on capacity and requirements and access control. Aribus are still looking at the components and how they are deployed.  Internal work right now , not ready to run models. When they are ready to run multiple models they will need to move stable.  Decision on the "tobe " stable environment are still being made. Is it exlusive use for Airbus or is it shared.   Target is to create stable environment for Datacommons. Airibus usecase for stable environment to be determined Mikhails update

Discuss with Heather how to transition this to LF team. Eileen to discuss with Heather.

Hi @eoriorda, we should include @rynofinn and @MightyNerdEric in the planning.  If Red Hat team can provide documentation on how the environment is set up, we can determine what knowledge transfer is needed to support transition

@HeatherAck Develop a plan for the stable instance environment.
When will Airbus be fully operational and PRR would be the next up to migrate to the stable instance.

@HeatherAck to follow up with Matthieu, dependent on superset and tool versions

Task on hold until we have stable superset/tool versions; need final snapshot from Airbus as well - next step create plan and checklist along with criteria. Heather to schedule mtg with @caldeirav , @MichaelTiemannOSC , @redmikhail and @sostrades-matthieu-meaux

CL2 will serve as baseline for stable cluster; but timing to do so still too early (need libraries to be included in notebook (pre-reqs for pachyderm), need info removed (old version of python), need open metadata issues resolved, Trino upgrades, etc.). @HeatherAck to create pre-req checklist for stable cluster

@HeatherAck to work with @erikerlandson to finalize list

@HeatherAck to follow up with @erikerlandson on versions, etc

see also https://github.com/os-climate/os_c_data_commons/issues/234

closing dup of 234


Add Cargo Deny Configuration

Fixes #issue
PR checklist:

[ ] make succeeds (on *nix)
[ ] make reftests succeeds
[ ] tested examples with the following backends:

@kvark I'll add this to CI in a second, but it won't actually deny duplicate dependencies it will just display them in the log and deny crates with unallowed licenses. Is how we'd want it?

Bummer, looks like the GitHub action only runs on Linux. That should be fine for now I think, I'll fix it.

bors r+

Hmm, what happened? Bors got stuck or something it looks like.

bors retry

I think bors isn't able to merge certain changes to GitHub Actions, so that's probably why it's failing


fix FAST-12048, ts is not correct in sample non-backfill mode

earliest and latest ts in sample (not backfill mode) doesn't work
1 if earliest/latest is an absolute time, the events always have the timestamp that equals earliest
2 if earlist = -100d and latest = now, the events have the timestamp between -200d and -100d

Hi. Looks good overall. Were you able to test out other scenarios? I would recommend testing that with bunch of sample scenarios provided to validate. Let me know how that looks!

Hi. Looks good overall. Were you able to test out other scenarios? I would recommend testing that with bunch of sample scenarios provided to validate. Let me know how that looks!

thanks tony. Just tried most samples in tests. It works well, but replay sometimes doesn't work.
it is a simple code issue, fixed in this PR


format is confusing when output to the CLI is redirected / piped

When redirecting the output from the CLI to less, the report includes control characters -- is there a working mode which does not not display them?
Fortitude indicates each spot which should be improved further, including a reference to the corresponding rule.  Compared to the report straight to the CLI, If I pipe this output to less by
fortitude check minimal.f90 | less

(because the then available vim navigation by j, k, gg, G potentially in parallel to a second pane), additional characters show up.  This is more difficult to read, than the original version:

This observation appears to be consistent with a redirect into a record e.g. by fortitude check minimal.f90 > out.txt, though I do not intend to use the checker this way.
Setup: Linux Debian 13/trixie with less (643), GNU bash (5.2.32), xfce4-terminal 1.1.3 (Xfce 4.18); and  fortitude (0.4.0, fetched from PyPI).
minimal.f90.txt

Thanks for the bug report @nbehrnd! This is fixed in #117, which will be in the next release.

@ZedThree Thank you for the hint.  With a bookmark for the entry on PyPI added; if the update is added there .and. I can observe locally the issue were gone, I'll close this this thread.


Review request for Microsoft.DBforPostgreSQL to add version preview/2024-11-01-preview

ARM (Control Plane) API Specification Update Pull Request
PR review workflow diagram
Please understand this diagram before proceeding. It explains how to get your PR approved & merged.

Purpose of this PR
What's the purpose of this PR? Check the specific option that applies. This is mandatory!

[ X] New API version for an existing resource provider. (If API spec is not defined in TypeSpec, the PR should have been created in adherence to OpenAPI specs PR creation guidance).

Due diligence checklist
To merge this PR, you must go through the following checklist and confirm you understood
and followed the instructions by checking all the boxes:

[X ] I confirm this PR is modifying Azure Resource Manager (ARM) related specifications, and not data plane related specifications.
[ X] I have reviewed following Resource Provider guidelines, including
ARM resource provider contract and
REST guidelines (estimated time: 4 hours).
I understand this is required before I can proceed to the diagram Step 2, "ARM API changes review", for this PR.

Additional information

 Viewing API changes

Index tuning
Added source types for migration
PG 17 support
UltraSSD_LRS storage type support
System assign identity
Citus on Flex cluster support

Suppressing failures
If one or multiple validation error/warning suppression(s) is detected in your PR, please follow the
suppressions guide to get approval.

Getting help

First, please carefully read through this PR description, from top to bottom. Please fill out the Purpose of this PR and Due diligence checklist.
If you don't have permissions to remove or add labels to the PR, request write access per aka.ms/azsdk/access#request-access-to-rest-api-or-sdk-repositories
To understand what you must do next to merge this PR, see the Next Steps to Merge comment. It will appear within few minutes of submitting this PR and will continue to be up-to-date with current PR state.
For guidance on fixing this PR CI check failures, see the hyperlinks provided in given failure
and https://aka.ms/ci-fix.
For help with ARM review (PR workflow diagram Step 2), see https://aka.ms/azsdk/pr-arm-review.
If the PR CI checks appear to be stuck in queued state, please add a comment with contents /azp run.
This should result in a new comment denoting a PR validation pipeline has started and the checks should be updated after few minutes.
If the help provided by the previous points is not enough, post to https://aka.ms/azsdk/support/specreview-channel and link to this PR.

/azp run

/azp run

@nasc17 - Please investigate required check failures (LintDiff, ModelValidation, SemanticValidation). Those need to be fixed or suppressed (only in cases where you don't think fixing is the right approach).

/azp run

reason: FlexibleServers namespace is already returning 202 as response for PUT. These APIs are under same namespace and hence keeping it as-is.

There are different error codes, but a same message is used. Please update the reasons based on the error code.

Refers to: specification/postgresql/resource-manager/readme.md:94 in 1b0d5b9. [](commit_id = 1b0d5b959c8a855535a9306a5c5d001d4730f6a8, deletion_comment = False)

/azp run

Please look at the open comments and address them

Please look at the open comments and address them

All have been addressed

/azp run

Could you please check https://github.com/Azure/azure-rest-api-specs/pull/30718/files#r1800240095 and Also fix the Swagger Avocado error

Bumped the comment that got resolved but is not implemented. Could you check that and Also fix the Swagger Avocado error

@ramoka178 The swagger avocado error we have been given exceptions for approval in previous request: https://github.com/Azure/azure-rest-api-specs/pull/30737/checks?check_run_id=30717134193

/azp run

@tadelesh SDK for go is succeeding now, but tag has not been removed.

Sending reminder


test suite leaks ssh-agent processes

The test suite starts ssh-agent in order to facilitate some functional testing.  Somehow it fails to clean these up and they remain running even after the test suite has completed.

We are moving our development planning to JIRA. This issue is now being tracked at https://clusterhq.atlassian.net/browse/FLOC-698. You are welcome to file additional issues in GitHub if that's easier for you.


azure deploy broken

error log:
Command: bash scripts/azuredeploy.sh
Invalid start-up command "chmod +x bin/slackin && ./bin/slackin" in package.json. Please use the format "node

I just tried running the Azure deploy. Didn't see the error message that you posted, but the Azure deploy no longer works because of the CAPTCHA (https://github.com/rauchg/slackin/pull/311). I see that some of the deployments (e.g. Heroku) were updated as a part of that PR, but Azure wasn't.
It would be great if it was optional (https://github.com/rauchg/slackin/issues/352), but we now need to configure the environment variables: $GOOGLE_CAPTCHA_SECRET $GOOGLE_CAPTCHA_SITEKEY.

I reread the error log and decided to act upon Invalid start-up command "chmod +x bin/slackin && ./bin/slackin" in package.json. Please use the format "node <script relative path>. That seems to be supported by this comment saying that sth might be wrong with script:start in package.json because the Kudo thingy reads that.
So I've changed that:
https://github.com/Code-for-All/slackin/blob/3ef754f1c8124a90ad301a3cba20b3621a6cc296/package.json#L62
But I'm still getting other errors:
Command: bash scripts/azuredeploy.sh
Using start-up script bin/slackin from package.json.
Node.js versions available on the platform are: 0.6.20, 0.8.2, 0.8.19, 0.8.26, 0.8.27, 0.8.28, 0.10.5, 0.10.18, 0.10.21, 0.10.24, 0.10.26, 0.10.28, 0.10.29, 0.10.31, 0.10.32, 0.10.40, 0.12.0, 0.12.2, 0.12.3, 0.12.6, 4.0.0, 4.1.0, 4.1.2, 4.2.1, 4.2.2, 4.2.3, 4.2.4, 4.3.0, 4.3.2, 4.4.0, 4.4.1, 4.4.6, 4.4.7, 4.5.0, 4.6.0, 4.6.1, 4.8.4, 5.0.0, 5.1.1, 5.3.0, 5.4.0, 5.5.0, 5.6.0, 5.7.0, 5.7.1, 5.8.0, 5.9.1, 6.0.0, 6.1.0, 6.2.2, 6.3.0, 6.5.0, 6.6.0, 6.7.0, 6.9.0, 6.9.1, 6.9.2, 6.9.4, 6.9.5, 6.10.0, 6.10.3, 6.11.1, 6.11.2, 6.11.5, 6.12.2, 6.12.3, 7.0.0, 7.1.0, 7.2.0, 7.3.0, 7.4.0, 7.5.0, 7.6.0, 7.7.0, 7.7.4, 7.10.0, 7.10.1, 8.0.0, 8.1.4, 8.4.0, 8.5.0, 8.7.0, 8.8.0, 8.8.1, 8.9.0, 8.9.3, 8.9.4, 8.10.0, 8.11.1, 10.0.0, 10.6.0.
Selected node.js version 6.11.1. Use package.json file to choose a different version.
Unable to locate npm version 3.3.12
An error has occurred during web site deployment.
select node version failed
Unable to locate npm version 3.3.12\r\nD:\Program Files (x86)\SiteExtensions\Kudu\79.11121.3655\bin\Scripts\starter.cmd bash scripts/azuredeploy.sh

I've managed to run it via docker container on Azure.. but that doesn't solve the problem.

Some googling around led to:

someone threw out the select node version code :P


Default to JRE certificate stores when using SSL and Load Balancing with Netty Client

If a LoadBalancingHttpClient is built with the "Secure" flag enabled to turn on SSL but without a trust store or keystore then in LoadBalancingRxClient the sslContextFactory is left as null.  This results in http being used in the protocol on http requests; despite originally requesting SSL.  It would be nice to default to the certs configured in the JRE if no key or trust store are configured or, at the least, throw an exception for a misconfiguration rather than silently change behavior.
The underlying objects that are used by LoadBalancingRxClient to support SSL (URLSslContextFactory and AbstractSslContextFactory) do default to the JRE certificate stores.  However, the logic in LoadBalancingRxClient that prevents the sslContextFactory from being instantiated disables this functionality.

Are there any known workarounds to this issue?

Yes, you can generate a dummy keystore:
keytool -genkey -keyalg RSA -keystore dummy-keystore.jks -keysize 2048

Leaving all default values except for the password.
Then setup Ribbon like this:
DefaultClientConfigImpl.getClientConfigWithDefaultValues()
            .set(CommonClientConfigKey.IsSecure, true)
            .set(CommonClientConfigKey.IsClientAuthRequired, false)
            .set(CommonClientConfigKey.KeyStore, "/path/to/dummy-keystore.jks")
            .set(CommonClientConfigKey.KeyStorePassword, "$keystorePassword")


[HELP] Wiki for Implementing custom dialog fields for Multifields

Hi @marcinczeczko,
Currently Wiki page has documentation for  Implementing custom dialog fields (TEXTFIELD). Can you please provide documentation for custom Multi Field? Appreciated.

Hi @DeChrish,
what kind of custom Multifield are you trying to implement?
In case of the current TouchUI module you can take a look at two classes that are used to handle this type of field (due to their nature, they are a bit more complex than regular fields):

Multifield
MultifieldItem

The rules are the same here - your custom classes should implement the DialogField interface and should be registered in your module, as described on the Wiki.
Let me know if you have any more questions.

@mkrzyzanowski Thanks, am working on it.  Sample YAML file will be helpful.

Multifield
MultifieldItem

Example YAML can be found on wiki here: https://github.com/Cognifide/bobcat/wiki/Component-configurer#complex-fields
Any custom field will follow the same rules, with the type matching its name.


Password LDAP seems not working

Hello,
When I note ldap_search_password and I check connectivity, I get this log in Harbor :
2021-03-05T12:35:44Z [ERROR] [/lib/http/error.go:54]: {"errors":[{"code":"INTERNAL_SERVER_ERROR","message":"LDAP connect fail, error: LDAP Result Code 206 "Empty password not allowed by the client": ldap: empty password not allowed by the client"}]}

Version 1.2.2

Can you please provide the terraform code for this?