modeloperator initial check-in

In order not to be intrusive, I added the model operator under the components folder. To avoid using nesting of modules, perhaps the root level package could be moved under the components folder as well.

hi @hxhp , thank you for your work. There are two mainly problem about this pr:

What's the relation between the model crd of this model-operator and https://github.com/kubeedge/sedna/blob/main/docs/proposals/dataset-and-model.md#dataset-and-model? Can the former just replace the later?
This pr uses kube-builder, and its mainly code structure is so many common with the root package. I'm inclined to merge these two into one now.

hi @hxhp , thank you for your great work. There are two mainly problem about this pr:

What's the relationship between the model crd of this model-operator and this project model crd? Can the former just replace the later?
This pr uses kube-builder, and its mainly code structure is so many common with the root package. I'm inclined to merge these into one in current phase.

Hi @llhuii , thank you for reviewing the PR. Please see my comments below:

The architecture of the model operator is different from the current root level package. Please see the architecture diagram here. Generally, there is no LC (local controller) in the design of this PR. The whole model operator is running in the cloud, and I will make it be deployed as a Kubernetes Deployment later. (Note that it doesn't/shouldn't run on the same node as cloudcore, actually it does not talk to KubeEdge cloudcore at all. It only talks to K8s API server.) The API provided by the model operator is the InferenceModel CRD itself. Higher level operators (joint inference and etc) just create an CRD instance, and model operator will create appropriate K8s resources (jobs, pods, deployments, services) as needed.
Honestly I don't think it's a good idea to merge the PR with the root level package. I think we should decouple them into separate microservices, and only use the CRD to interact with each other. And this is why I am thinking of having a components directory, where each sub directory hosts a single service. And this way we don't have to handle submodules (nesting of modules).

Please let me know what you think. Thanks.

Higher level operators (joint inference and etc.) also just list and watch k8s master, like kube-controller-manager we can control these sedna operators in a binary, one microservice instead of separate microservices.

Higher level operators (joint inference and etc.) also just list and watch k8s master, like kube-controller-manager we can control these sedna operators in a binary, one microservice instead of separate binary/microservices.

Yeah I agree that a controller manager can manage more than one controllers. I am also trying to let the model operator to manage both InferenceModel and TraningModel controllers. But it looks to me that joint inference, federated learning and incremental learning are different scenarios and they have quite different operating logic. I am worried about scenarios like: Let's say we already deployed v0 of sedna in a production environment. Now we have a code change in federated learning, be it bug fix or feature improvement, and we want to deploy this change. If all these operators are in a single code base, deployed as a single binary, then all the other operators will have to be restarted. If there's a serious bug in this change, the single monolithic binary may crash, tearing down the whole system, instead of a single operator. (blast radius)

We need to balance this.

But it looks to me that joint inference, federated learning and incremental learning are different scenarios and they have quite different operating logic.

Yes, different operating logic with their own implentation go file or package, but they all belongs to Sedna.
We want simplicity for our small team at the beginning phase, single level package is simple and easy from my view.
Is sedna code base big/complicated enough for splitting into multi micro-servis?  As I can see, it is less complicated than kube-controller-manager.
And even into multi micro-services with different kube-build generated subdirectory, hard to maintain these independent but very common files e.g. go.mod/hack/makefile.
And you suggest for splitting, what's the rule of splitting?

Different scenario with different micro-service? then there would be many micro-services to maintain.

Let's say we already deployed v0 of sedna in a production environment. Now we have a code change in federated learning, be it bug fix or feature improvement, and we want to deploy this change. If all these operators are in a single code base, deployed as a single binary, then all the other operators will have to be restarted.

If the users want to deploy new feature or bug, then just replace the old version with the new-released version by editing sedna deployment file. They really don't want to manitain multi versions of a package.

then all the other operators will have to be restarted.

So what? then they can list-watch again.

If there's a serious bug in this change, the single monolithic binary may crash, tearing down the whole system, instead of a single operator.
Then let's our release to be stable.
Same question for kube-controller-manager, k8s, docker, linux kernel, what will they do for a bug fix?

I think we should focus on how to locate sedna. Is sedna a collection of edge AI operators or a managers of edge AI operators? I think should be the latter. Different edge AI operators have common characteristics, not isolated ones. For example, they all need to list-watch model resources and pod resources, the SharedInformer can help us to reduce pressure on k8s if all the edge AI operators run in on process.

Let's say we already deployed v0 of sedna in a production environment. Now we have a code change in federated learning, be it bug fix or feature improvement, and we want to deploy this change. If all these operators are in a single code base, deployed as a single binary, then all the other operators will have to be restarted.

I think in a production enviroment, we can handle this with full testing and gray deployment.


Upd/neo-go v0.99.2

Closes #327.
Stricly speaking, it did not contain deprecated methods (at least for for v0.99.2) but v0.99.2 really simplifies the code.

Neo-go v0.99.2 requires go1.17+ (//go:build directive) so I updated to v1.17 and also added v1.19 tests.


Timezone detection, Y2k36 handling, code refactoring.

I implemented a function to fetch timezone information from an online geolocation service.
At first I wanted it to be generic, to work with a multitude of geolocation services, but I couldn't get libcurl to work under WUPS, it always crashed. So I wrote a bare-bones HTTP (no HTTPS) client, and looked for a free service that worked over plain HTTP, and found one: ip-api.com. Even better, it can produce results in plain text, one per line, so there's no need to parse JSON or XML content.
To use the feature, go to the "Configuration" screen, select the "Detect Timezone" option, and press "A". The geolocation service will look up the IP into a database, and report back what it thinks is the timezone name and its corresponding UTC offset. That UTC offset is then used to update the "Hours" and "Minutes" offsets.
This is not done automatically, because I didn't want to introduce privacy concerns (even though no identifiable information is transmitted, only a HTTP GET request with "User-Agent: Wii U Time Sync Plugin".)
Most users will ever need to use this feature twice per year, to adjust for DST changes. I did not attempt to use the timzeone database to automate DST changes, because it's more work than it's worth it; it's updated multiple times per year, it's obnoxious to parse, and any sort of automated downloading of that DB would require libcurl (and again, I could not get it to work inside WUPS.)
Also, I added code to handle the NTP wraparound, that will happen in 2036. Hopefully we get NTPv5 before then.
Most of the changes are code cleanup and refactoring. The core functionality (performing NTP queries) now lives in core.cpp/core.hpp. The "Configuration" and "Preview" screens are setup in their own separate files. Configuration variables are in the cfg.cpp/cfg.hpp files. Helper functions and classes, that didn't seem worth moving to their own files, are all in utils.cpp/utils.hpp. The main.cpp file has been reduced to do only WUPS plugin stuffs. I think this makes the code overall easier to understand and modify.

Looks and works good on my end, no issues! As usual, thank you for these changes, they really make the plugin a whole lot better. I appreciate the timezone fetching for example, I think that's a super cool idea and executed well (thank you for making it optional!)
I'm probably going to make a new release when I'm free this weekend, and both of your recent pull requests will be apart of that.


[Suggestion] Embed Resources/DLL's within the app to make it portable

If you use something like Costura.Fody (https://github.com/Fody/Costura) to embed the resources within the EXE itself, it makes it a self-contained application (slightly bloats the EXE, but then end users don't have to keep track of DLL's if they were to move, update, or delete it).
Have done this with one of my own WPF apps, and it seemed to work well.

Right, i will do this for the next release.


Api service and Web App fail to start

Type of issue
Code doesn't work
Description
[Enter feedback here]
After completing the deployment, API service and Web App crash. The logs on Azure reveal for both components the error message:
You must install or update .NET to run this application.

It is not obvious which of the 100 steps must be adjusted to fix this failure.
Page URL
https://learn.microsoft.com/en-us/dotnet/aspire/deployment/azure/aca-deployment?tabs=visual-studio%2Cinstall-az-windows%2Cpowershell&pivots=azure-bicep
Content source URL
https://github.com/dotnet/docs-aspire/blob/main/docs/deployment/azure/aca-deployment.md
Document Version Independent Id
1362fc62-686d-2b3f-c139-7725fc8dca5f
Article author
@IEvangelist

Tagging @bradygaster on this one.

@alexwolfmsft - do you know what this might be related to? Have you experienced it before?

Hi @Monte-Christo - could you try to upgrade to preview-3 and report back if you're able to reproduce this.

Well, it took some time (I have a day job) but I was able to update all components and try again: no joy. This is my version of the Aspire SDK:
aspire                     8.0.0-preview.4.24156.9/8.0.100      VS 17.10.34707.107
After deploying to Azure, I am getting the same error as before (see log entry attached). Maybe it is time to do some real investigating?

Hi again @Monte-Christo - if you're still seeing this issue, I don't believe it's a docs issue, but rather a product issue. Preview 7 just released last week, and the product officially goes GA next week with //build. That means support too. Curious if you're still seeing the issue, if so, I'll transfer this to dotnet/aspire.


Add link for Trace migration guide in OpenCensus' docs

Migration guide being added here: https://github.com/census-instrumentation/opencensus-php/pull/90

@jdpedrie Just FYI, it's not published yet, but will be ASAP.

roger that!


CLI: Quote script arguments in error message

This makes it easier to copy-paste the command from the error message.

Thank you for the contribution!


How does shipyard integrate with docker-swarm?

It's my understanding that shipyard works with docker swarm. So my question is in docker 1.12 does shipyard need to be secured with TLS since docker does this by default? Also, can the shipyard controller be used as a standalone interface for docker swarm? Many thanks.
p.s. I'm new to github. Is it acceptable to ask these types of questions here? I couldn't decided where to ask.

Currently shipyard works with swarm but not yet swarm mode which is new in
docker 1.12. We are working on a branch to support this which would support
all of the built-in security once finished.
On Oct 19, 2016 8:52 PM, "Nokan Page"<EMAIL_ADDRESS>wrote:

It's my understanding that shipyard works with docker swarm. So my
question is in docker 1.12 does shipyard need to be secured with TLS since
docker does this by default? Also, can the shipyard controller be used as a
standalone interface for docker swarm? Many thanks.
p.s. I'm new to github. Is it acceptable to ask these types of questions
here? I couldn't decided where to ask.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/shipyard/shipyard/issues/889, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAP6IrIZQ4wxQqqM-bJ1nVnlb5ksrhUMks5q1rtcgaJpZM4KbnOc
.

Is there anything I could do to help out with this effort? Is there a list of targets somewhere?

Actually, I see that the front end doesn't display in Safari, so I think I'll start with that.

Created pull request #939

@cloutiertyler There's lots of work remaining on the swarm-mode branch that we'd love help with.  The simplest things are just fleshing out our "Create" and "Inspect" pages for each of the resources (Services, Containers, Volumes, Networks, etc.).
If you're interested in tackling something a bit different to that, there are a few items in: https://github.com/shipyard/shipyard/blob/swarm-mode/ui/todo.md
If you have any questions, feel free to give us a shout!

Okay, fleshing out the "Create" and "Inspect" pages sounds good. I noticed when I was using it that I had to hop over to command line for a lot of things, so I might try to work in some of the more important ones. I'll probably work on these first.
Also, I was thinking I might try integrating a component which is something like https://github.com/ManoMarks/docker-swarm-visualizer, just because it looks super fun. Not sure if that's what you guys are going for though lol.

Awesome, sounds good.  I just haven't had the spare time to map out the rest of the API, so that would be a huge bonus.  Thanks for your interest and help! :smile:
I think the plan at the moment is just get it back to a basic level of functionality, similar to the angular UI and then we can start adding bells and whistles :smile:.

It looks like most of the API (back end) is just proxied swarm APIs. So I think I'll just go through those adding UI to each of them. I'd like to bring it up to par with as much command line functionality as I can.


Errors when using orderBy()

When using the orderBy() method the following errors occur and get an infinite progress indicator on the UI.
#0      MethodChannelQuery.get (package:cloud_firestore_platform_interface/src/method_channel/method_channel_query.dart:110:7)
<asynchronous suspension>
#1      _JsonQuery.get (package:cloud_firestore/src/query.dart:390:9)
<asynchronous suspension>
#2      PaginationCubit.refreshPaginatedList (package:paginate_firestore/bloc/pagination_cubit.dart:61:29)
<asynchronous suspension>

However, after removing the orderBy() method, everything works fine. I've created index's for the collection and can confirm the respective fields exist.
This is how I'm implementing it:
class PostFeed extends StatelessWidget {
  const PostFeed(
      {required this.documentSnapshot, required this.header, Key? key})
      : super(key: key);
  final Query<Object?> documentSnapshot;
  final Widget header;

  @override
  Widget build(BuildContext context) {
    PaginateRefreshedChangeListener refreshChangeListener =
        PaginateRefreshedChangeListener();
    return RefreshIndicator(
      child: PaginateFirestore(
        header: SliverToBoxAdapter(
            child: Column(children: [header, const SizedBox(height: 150)])),
        // isLive: true,
        shrinkWrap: true,
        itemsPerPage: 3,
        query: documentSnapshot.orderBy('title'),
        listeners: [refreshChangeListener],
        itemBuilderType: PaginateBuilderType.listView,
        physics: const AlwaysScrollableScrollPhysics(),
        itemBuilder: (context, documentSnapshots, index) {
          final data = documentSnapshots[index];
          return Card(
              child: SizedBox(height: 300.9, child: Text(data.get('title'))));
        },
      ),
      onRefresh: () async => refreshChangeListener.refreshed = false,
    );
  }
}

Loving this package, it literally saved my ass, but this was a bit of a bummer. Any ideas?
Thanks

Apologies, operator fault here. This can be totally ignored but for those wondering - This is what happens when you stack the orderBy() mehtod therefore, just use one.


Add free-form text to survey module

https://github.com/PAIR-code/llm-mediation-experiments/commit/c5480afed905b00f78ced2fd72950fcd5d582216
https://github.com/PAIR-code/llm-mediation-experiments/commit/b36875c705bac439cd2532bb2b90f5cd2ded4e8a


Error deploying my lambda function with the plugin: FUNCTION_ERROR_INIT_FAILURE

Hi there! I'm getting the following error when I try to deploy my function with the plugin
Serverless Error ---------------------------------------

An error occurred: AppProvConcLambdaAlias - Provisioned Concurrency configuration failed to be applied. Reason: FUNCTION_ERROR_INIT_FAILURE

Any idea on how to debug this error?

Hi, @tofredi this is common when your code has an issue and lambda cannot initialize correctly. If you remove the plugin, but leave provisioned concurrency, you will see the same error.

Yes, I have just fixed the error
Thanks @claydanford

Removing plugin, actually, solves this problem.


Changes PullRequest Index to ForgeRemoteID type

This allows using string as pull requst ID in the forges. Also, provides a uniform type for remote forge IDs among different objects.
Resolves #2746
Needed by #2355

PS: we are currently in feature freeze, until v2.0.0 is tagged :)

PS: we are currently in feature freeze, until v2.0.0 is tagged :)

Yes, I am aware about it and looking forward for the release!  🥳
This can definitely wait!


How to change Preview ratio in CameraXBasic

I want to change the ratio of preview & capture image.
i inserted this code but not worked.
var screenAspectRatio = AspectRatio.RATIO_4_3
preview = Preview.Builder()
// We request aspect ratio but no resolution
.setTargetAspectRatio(screenAspectRatio)
// Set initial target rotation
//.setTargetResolution(Size(1440, 1920))
.setTargetRotation(rotation)
.build()
Preview is always 16:9...
Is there anything I missed?

viewFinder.layoutParams.width = requireContext().resources.displayMetrics.widthPixels
viewFinder.layoutParams.height = viewFinder.layoutParams.width * 4 / 3
It is worked!


Add groundwork for tool management

Add the features and utilities for upcoming generators and executors that require .NET tools.
Fixes #44

@bcallaghan-et In testing this at home I ran into the init schematic failing when calling it via nx g @nx-dotnet/core:init, unless it had already been called by a different schematic (i.e. the app generator). This is because nx generators are always called with two parameters by the CLI. Host, and options.
Our init schematic currently does not take any options, but Nx still passes it.  Since we need to add a default parameter, we also have to add an unused second parameter (I recommend just using an _ for it for now, we can type it and change its name to options later if we need them.

The unit tests do not catch this since we are importing the generator and calling it directly. E2E tests should catch this though. I am working on getting the e2e in better shape, I do not like how it works currently.


crypto/x509: RSA-PSS signatures on certificates should be encoded without optional NULLs

What version of Go are you using (go version)?

$ go version
go version go1.12.15 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/lib/golang"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build545125816=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Sign a certificate with RSASSA-PSS
What did you expect to see?
The AlgorithmId does not include the two optional NULL fields, which matches the behavior of OpenSSL and conforms to a SHOULD in RFC8017.
What did you see instead?
The AlgorithmId does include the two optional NULL fields, which does not match the behavior of OpenSSL and does not conform to a SHOULD in RFC8017.

@katiehockman @FiloSottile

If this is both a SHOULD in the RFC and the deployed behavior of OpenSSL, we should just do it, yeah.

@FiloSottile The (trivial) fix is to remove the asn1.NullRawValue initializers in the RSA-PSS AlgorithmId serialization code.

Not so fast!  This turns out to violate https://github.com/mozilla/pkipolicy/blob/master/rootstore/policy.md#511-rsa which requires the NULL parameters.

This is a long running problem, about which RFC 4055 captures some of the context:

There are two possible encodings for the AlgorithmIdentifier
parameters field associated with these object identifiers.  The two
alternatives arise from the loss of the OPTIONAL associated with the
algorithm identifier parameters when the 1988 syntax for
AlgorithmIdentifier was translated into the 1997 syntax.  Later the
OPTIONAL was recovered via a defect report, but by then many people
thought that algorithm parameters were mandatory.  Because of this
history some implementations encode parameters as a NULL element
while others omit them entirely.  The correct encoding is to omit the
parameters field; however, when RSASSA-PSS and RSAES-OAEP were
defined, it was done using the NULL parameters rather than absent
parameters.

RFC 8017 somewhat complicates this by then seemingly saying in the appendix ASN.1 module (which is... rather confusing) that omitting the parameters for OAEP-PSSDigestAlgorithms members is acceptable. Given there are a multitude of 'acceptable' encodings the web PKI has tried to reduce complexity by picking a single valid encoding, and enforcing it via root program requirements.

This is a long running problem, about which RFC 4055 captures some of the context:

There are two possible encodings for the AlgorithmIdentifier
parameters field associated with these object identifiers.  The two
alternatives arise from the loss of the OPTIONAL associated with the
algorithm identifier parameters when the 1988 syntax for
AlgorithmIdentifier was translated into the 1997 syntax.  Later the
OPTIONAL was recovered via a defect report, but by then many people
thought that algorithm parameters were mandatory.  Because of this
history some implementations encode parameters as a NULL element
while others omit them entirely.  The correct encoding is to omit the
parameters field; however, when RSASSA-PSS and RSAES-OAEP were
defined, it was done using the NULL parameters rather than absent
parameters.

RFC 8017 somewhat complicates this by then seemingly saying in the appendix ASN.1 module (which is... rather confusing) that omitting the parameters for OAEP-PSSDigestAlgorithms members is acceptable. Given there are a multitude of 'acceptable' encodings the web PKI has tried to reduce complexity by picking a single valid encoding, and enforcing it via root program requirements.

So is this an OpenSSL bug rather than a Go bug?

Eh, it kind of depends. OpenSSL isn't really doing anything wrong, it is totally standards compliant to omit the parameters field, but if someone is using OpenSSL to generate certificates that are intended to be publicly trusted under roots included in the Mozilla root program then they are violating the store policy.

Eh, it kind of depends. OpenSSL isn't really doing anything wrong, it is totally standards compliant to omit the parameters field, but if someone is using OpenSSL to generate certificates that are intended to be publicly trusted under roots included in the Mozilla root program then they are violating the store policy.

Should Mozilla’s policy be changed?

I don't think so, no. The choice of the 4055 encoding was explicitly made to reduce the possible valid encodings to one.

Circling back to this, I think it makes sense to document why this choice is made, and probably close this out as a wont-fix.

Thank you for investigating. crypto/x509 targets primarily the WebPKI, so indeed we should follow the root stores policies.


How to detect target element when it zoomed via function onBeforeOpen/onBeforeClose

Hello,
I've read your documentation and I can not get automatically selector ID of target photo when it zoomed. I want to manipulate with it by onBeforeOpen/onBeforeClose function.
How to get target photo ID via function or something else?
Thanks

I found a solution
var zooming = new Zooming({           //... 	  onBeforeOpen: function(target){ 		console.log(target); 	  },         //... });

Yes, you will have the target as callback argument.


Unsafe load() call disabled by Gentoo. See bug #659348

I have installed tuijam just now in my Gentoo system and upon the first run I got the error message. Full error:
λ ~/ tuijam
starting up.
logging in.
Traceback (most recent call last):
  File "/home/guitmz/.local/bin/tuijam", line 1363, in <module>
    app = main(debug=debug)
  File "/home/guitmz/.local/bin/tuijam", line 1328, in main
    if not app.login():
  File "/home/guitmz/.local/bin/tuijam", line 692, in login
    credentials = self.read_config()
  File "/home/guitmz/.local/bin/tuijam", line 706, in read_config
    config = yaml.load(f.read())
  File "/usr/lib64/python3.6/site-packages/yaml/__init__.py", line 109, in load
    raise RuntimeError("Unsafe load() call disabled by Gentoo. See bug #659348")
RuntimeError: Unsafe load() call disabled by Gentoo. See bug #659348

Relevant Gentoo bug report https://bugs.gentoo.org/659348
It looks trivial to fix this issue (but I haven't really tried yet). Replacing yaml.load() calls with yaml.safe_load() should do it. Let me know your thoughts on this one.
Thanks

I also recommend replacing yaml.dump() with yaml.safe_dump()


Allow "first name" to correctly handle names in "Lastname, Firstname" format

Updates the loggedInUserFirstname  variable to handle fullnames in both formats: "Firstname Lastname" and "Lastname, Firstname"

Thanks, @meschwartz!


apache-superset 0.0.0.dev0 requires sqlalchemy!=1.3.21

A clear and concise description of what the bug is.
How to reproduce the bug
pip install clickhouse-sqlalchemy
ERROR:
Collecting clickhouse-sqlalchemy
Downloading clickhouse-sqlalchemy-0.2.2.tar.gz (35 kB)
Collecting sqlalchemy<1.5,>=1.4
Downloading SQLAlchemy-1.4.41-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.6 MB)
|████████████████████████████████| 1.6 MB 20 kB/s
Requirement already satisfied: requests in /usr/local/lib/python3.8/site-packages (from clickhouse-sqlalchemy) (2.26.0)
Collecting clickhouse-driver>=0.1.2
Downloading clickhouse_driver-0.2.4-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (732 kB)
|████████████████████████████████| 732 kB 24 kB/s
Requirement already satisfied: pytz in /usr/local/lib/python3.8/site-packages (from clickhouse-driver>=0.1.2->clickhouse-sqlalchemy) (2021.1)
Collecting tzlocal
Downloading tzlocal-4.2-py3-none-any.whl (19 kB)
Collecting greenlet!=0.4.17
Downloading greenlet-1.1.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (157 kB)
|████████████████████████████████| 157 kB 28 kB/s
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.8/site-packages (from requests->clickhouse-sqlalchemy) (2.0.4)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.8/site-packages (from requests->clickhouse-sqlalchemy) (3.2)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/site-packages (from requests->clickhouse-sqlalchemy) (1.26.6)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.8/site-packages (from requests->clickhouse-sqlalchemy) (2021.5.30)
Collecting pytz-deprecation-shim
Downloading pytz_deprecation_shim-0.1.0.post0-py2.py3-none-any.whl (15 kB)
Collecting backports.zoneinfo
Downloading backports.zoneinfo-0.2.1-cp38-cp38-manylinux1_x86_64.whl (74 kB)
|████████████████████████████████| 74 kB 72 kB/s
Collecting tzdata
Downloading tzdata-2022.2-py2.py3-none-any.whl (336 kB)
|████████████████████████████████| 336 kB 59 kB/s
Building wheels for collected packages: clickhouse-sqlalchemy
Building wheel for clickhouse-sqlalchemy (setup.py) ... done
Created wheel for clickhouse-sqlalchemy: filename=clickhouse_sqlalchemy-0.2.2-py3-none-any.whl size=46945 sha256=b3e81a6ffaee6df786eebaba7274211aa1d703def2d4a8c401791e360c476fa4
Stored in directory: /root/.cache/pip/wheels/fb/39/b2/82f886ce2e5df1ed5e28230890c6f28bac9faac69e429f399f
Successfully built clickhouse-sqlalchemy
Installing collected packages: tzdata, backports.zoneinfo, pytz-deprecation-shim, tzlocal, greenlet, sqlalchemy, clickhouse-driver, clickhouse-sqlalchemy
Attempting uninstall: sqlalchemy
Found existing installation: SQLAlchemy 1.3.24
Uninstalling SQLAlchemy-1.3.24:
Successfully uninstalled SQLAlchemy-1.3.24
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flask-appbuilder 3.4.1 requires SQLAlchemy<1.4.0, but you have sqlalchemy 1.4.41 which is incompatible.
apache-superset 0.0.0.dev0 requires sqlalchemy!=1.3.21,<1.4,>=1.3.16, but you have sqlalchemy 1.4.41 which is incompatible.
Successfully installed backports.zoneinfo-0.2.1 clickhouse-driver-0.2.4 clickhouse-sqlalchemy-0.2.2 greenlet-1.1.3 pytz-deprecation-shim-0.1.0.post0 sqlalchemy-1.4.41 tzdata-2022.2 tzlocal-4.2
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour withthe system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 22.2.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

clickhouse-sqlalchemy 0.2.2 requires SqlAlchemy 1.4, while Superset up to 2.0.0 requires SQLAlchemy 1.3.  I believe the latest Superset development version now supports SqlAlchemy 1.4, so one approach is to pull the latest Superset Github or docker version.
You can also try pinning clickhouse-sqlalchemy to 0.2.0 or 0.2.1.
Finally please consider https://github.com/ClickHouse/clickhouse-connect for Superset with ClickHouse, especially if you want HTTP compatibility (proxy/VPN friendly).  Full disclosure, I'm the primary author.

Since we're now on 3.1, on the verge of 4.0, we're not supporting 2.x  or prior versions actively. If anyone is experiencing this issue in current versions, please file a new issue with updated context and repro steps. Thank you for understanding :)


Could you specify the number of an object?

Occured to me that there should be good enough ML tooling to get quite good results by stating something like.
"3 cars"
Grounding DIno proceeds to run inference with high sensitivity to cars before keeping the top 3 preds. Anyone know of any demos doing this?
Or is there already a computer vision model that would cope directly with this sort of prompt?

There's no guarantee of the model's outputs, since the model gives probabilities only. But you can choose the top 3 objects as the final output.


Allow PyDataFrame to be used from other projects

Which issue does this PR close?

Closes https://github.com/apache/arrow-datafusion-python/issues/581
Rationale for this change

I would like other projects (such as PyBallista) to be able to re-use the PyDataFrame struct.
See https://github.com/apache/arrow-ballista/pull/976 for an example of how this is leveraged.
What changes are included in this PR?

Make some things public.
Are there any user-facing changes?

@jdye64 fyi


How do I get the ± values in my experimental results?

Hi, bothering you again~
For example, in the wine quality dataset, the ensemble performance is 0.620±0.007, and it is known that go.py will get three sets of scores, so how to get this ±0.007?

Hi, you will have to compute it manually. For example, this is how you can compute the mean and the standard deviation of the score of MLP on california (the same applies to ensembles and all other algorithms):
import json
import statistics
from pathlib import Path

scores = [
    json.loads(x.joinpath('report.json').read_text())['metrics']['test']['score']
    for x in Path('exp/mlp/california/0-evaluation').iterdir()
    if x.is_dir()
]
print(f'{statistics.mean(scores)} +- {statistics.stdev(scores)}')

OK! Thanks


Fix scroller detection in angular directive

Fixes #246

@norkunas probably I'm not alone with this issue. But I found #248 more useful in that case.

@atsoy 👍

Excuse the delay!
Can I ask, why the timeout? Is this still relevant if/when #248 gets merged?

@WickyNilliams I don't remember now, but it fixed my issue. About #248 i think it's less optimal to watch for the scroller than to get it in a timeout

@norkunas -- I am unable to get headroom.js to work with our angular environment; and, since you  seem to one the "resident angular implementers" of this library - I though I would reach out to you.  I have a note out to @WickyNilliams, but he says that he's not an expert at Angular.
The following has been done so far, but for the life of me cannot figure out why it is not working or what is wrong with my implementation:

Added headroom.js
Added angular.headroom.js
Added reference to index.html

Added requirement to angularJS module

angular.module('PSApp', [
  'fcsa-number',
  'headroom',
  'typer',...
])

Added some basic CSS

.headroom {
    will-change: transform;
    transition: transform 200ms linear;
}

.headroom--pinned {
    transform: translateY(0%);
}

.headroom--unpinned {
    transform: translateY(-100%);
}

What am I doing wrong?  It just doesn't want to work... activate.  I scroll up/down and nothing happens...
Is this still a supported plugin (++ for, Angular).  Our site is developed using angular 1.2.x.

@tabish89 I can't say what's wrong only with code without live example (html code included would shouw more about your usage), but I suggest to check the element when scrolling if the classes are applied at all. Maybe scroller="#scroll-container" is missing in your code if you use it as SPA and the scrolling container is not the document.body.

@toddmotto
My bad.  I don't want to be presumptuous about how much time you might have to help me, so let me begin by saying that I am extremely appreciative of your goodwill.  I'm a coding novice, so your patience is also much appreciated in advance.
In addition to the foregoing code snippets shared, here is an example of how I'm referencing headroom.js in HTML.  If you prefer, I can see how I can share our code base but that wouldn't let you set up a live example easily because there are a number of bespoke dependencies required in order to load the web application.
The 'codepen' most similar to how headroom.js is implemented on our end can be found [https://codepen.io/frankcastro/pen/KzLLpr?page=1&] here.  I obtained the library from NPM and have compared it to other versions on the web and:

headroom.js is the same
angular-headroom.js is the same; hwoever, I have seen another version out there (not merged) with a $timeout function

I have not integrated a bespoke jQuery reference and was hoping to use the default settings built in Angular (I believe there is one).

<headroom tolerance="0" offset="0" classes="{pinned:'headroom-pinned',unpinned:'headroom-unpinned',initial:'headroom'}">
TEST HEADROOM

  <md-toolbar class="md-primary md-hue-3 header" layout="row" layout-align="start center" hide-sm hide-xs>

    ...

    <md-button class="md-icon-button" ng-click="toggleSideNav()" aria-label="SideNav">
       <md-icon md-svg-icon="menu" style="fill: white;"></md-icon>
    </md-button>

  </md-toolbar>

</headroom>

I checked CONSOLE (Google Chrome) to see whether the classes are being applied; and, they seem to be there.

.headroom-pinned {                global.css:47
   transform: translateY(-100%);
}
_Appears when scrolling up

.headroom-unpinned {            global.css:43
   transform: translateY(0%);
}
Appears when scrolling down

.headroom{                               global.css:38
   will-change: transform;
   transition: transform 200ms
       [x] linear;
}

Major headscratcher...

I'm using headroom directly on md-toolbar directive instead of wrapping it. Please try:
<md-toolbar headroom tolerance="0" offset="0" classes="{pinned:'headroom-pinned',unpinned:'headroom-unpinned',initial:'headroom'}" class="md-primary md-hue-3 header" layout="row" layout-align="start center" hide-sm hide-xs>

    ...
  <md-button class="md-icon-button" ng-click="toggleSideNav()" aria-label="SideNav">
    <md-icon md-svg-icon="menu" style="fill: white;"></md-icon>
  </md-button>
</md-toolbar>

Just tried your recommendation, but still not working.  :-(  Here is the whole HTML to make sure nothing is possibly conflicting with the headroom.js code:

<md-toolbar headroom tolerance="205" offset="5" classes="{pinned:'headroom-pinned',unpinned:'headroom-unpinned',initial:'headroom'}" class="md-primary md-hue-3 header" layout="row" layout-align="start center" hide-sm hide-xs>

  <md-button style="text-transform: none;" ng-if="!isLoggedIn">
    Sign In
  </md-button>

  <md-button style="text-transform: none;">
    Features
  </md-button>

  <md-button ng-if="!isLoggedIn" style="text-transform: none;">
    Pricing
  </md-button>

  <md-button ng-if="isLoggedIn" style="text-transform: none;">
    Subscription
  </md-button>

  <md-button style="text-transform: none;">
    Company
  </md-button>

  <md-menu ng-controller="SiteCtrl.Menu as ctrl">
    <md-button style="text-transform: none" ng-click="$mdMenu.open($event)">
      FAQ
    </md-button>
    <md-menu-content class="light-gray-background" width="3">
      <md-menu-item ng-repeat="item in ['General', 'Business', 'Technology']">
        <md-button ng-click="ctrl.selectItem($index, false)">

{{item}}

        </md-button>
      </md-menu-item>
    </md-menu-content>
  </md-menu>

  <md-button class="md-icon-button" ng-click="toggleSideNav()" aria-label="SideNav">
    <md-icon md-svg-icon="menu" style="fill: white;"></md-icon>
  </md-button>
</md-toolbar>

I saw some other variations on this on the web, where someone set it up as a FACTORY 'Directive' (I've tried this too, but not working)...
  .directive('headroom', function() {
    return {
      restrict: 'E',
      replace: true,
      transclude: true,
      template: '

',
      link: function(scope, element, attrs) {
        var headroom, options;

        var setupOptions = function() {
          if (options)
            return;

          options = [];
          angular.forEach(Headroom.options, function(value, key) {
            if (key == 'classes' && attrs[key])
              options[key] = angular.fromJson(attrs[key]);
            else
              options[key] = attrs[key] || Headroom.options[key];
          });

          if (attrs.onPin) {
            options.onPin = function() {
              scope.$apply(attrs.onPin);
            };
          }
          if (attrs.onUnpin) {
            options.onUnpin = function() {
              scope.$apply(attrs.onUnpin);
            };
          }
          if (attrs.onTop) {
            options.onTop = function() {
              scope.$apply(attrs.onTop);
            };
          }
          if (attrs.onNotTop) {
            options.onNotTop = function() {
              scope.$apply(attrs.onNotTop);
            };
          }
        };

        var initialize = function() {
          setupOptions();
          headroom = new Headroom(element[0], options);
          headroom.init();
        };

        var destroy = function() {
          if (headroom)
            headroom.destroy();
        };

        attrs.$observe('disabled', function(value) {
          if (angular.isUndefined(value))
            return;

          if (value === true)
            destroy();
          else
            initialize();
        });

        scope.$on('$destroy', function() {
          destroy();
        });
      }
    };
  });

I think you need to set the scroller="md-content" or to some other
selector which is your content with scrolling.
2017-08-24 16:16 "Tabish Rizvi"<EMAIL_ADDRESS>rašė:

Just tried your recommendation, but still not working. :-( Here is the
whole HTML to make sure nothing is possibly conflicting with the
headroom.js code:

Sign In

Features

Pricing

Subscription

Company

FAQ

{{item}}

I saw some other variations on this on the web, where someone set it up as
a FACTORY 'Directive' (I've tried this too, but not working)...
.directive('headroom', function() {
return {
restrict: 'E',
replace: true,
transclude: true,
template: '',
link: function(scope, element, attrs) {
var headroom, options;
    var setupOptions = function() {
      if (options)
        return;

      options = [];
      angular.forEach(Headroom.options, function(value, key) {
        if (key == 'classes' && attrs[key])
          options[key] = angular.fromJson(attrs[key]);
        else
          options[key] = attrs[key] || Headroom.options[key];
      });

      if (attrs.onPin) {
        options.onPin = function() {
          scope.$apply(attrs.onPin);
        };
      }
      if (attrs.onUnpin) {
        options.onUnpin = function() {
          scope.$apply(attrs.onUnpin);
        };
      }
      if (attrs.onTop) {
        options.onTop = function() {
          scope.$apply(attrs.onTop);
        };
      }
      if (attrs.onNotTop) {
        options.onNotTop = function() {
          scope.$apply(attrs.onNotTop);
        };
      }
    };

    var initialize = function() {
      setupOptions();
      headroom = new Headroom(element[0], options);
      headroom.init();
    };

    var destroy = function() {
      if (headroom)
        headroom.destroy();
    };

    attrs.$observe('disabled', function(value) {
      if (angular.isUndefined(value))
        return;

      if (value === true)
        destroy();
      else
        initialize();
    });

    scope.$on('$destroy', function() {
      destroy();
    });
  }
};

});
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/WickyNilliams/headroom.js/pull/247#issuecomment-324631625,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACmMOJdvxm1hzoEB7UvuGhRGukql6hIQks5sbXe8gaJpZM4J7bD7
.

Tried it both ways... simply doesn't want to work.
<headroom scroller="md-content" tolerance="205" offset="5" classes="{pinned:'headroom-pinned',unpinned:'headroom-unpinned',initial:'headroom'}">
TEST HEADROOM

  <md-toolbar class="md-primary md-hue-3 header" layout="row" layout-align="start center" hide-sm hide-xs>

...

</md-toolbar>
</headroom>

or:

```

Think I might have figured out "WHY" it is not working.  Here is a [https://codepen.io/tabish89/pen/eEjdrj] CodePen with a lite-working version of the code.
You will notice in the HTML that there is a md-menu embedded in my 'md-toolbar', which seems to be conflicting with the headroom code.  When it is disabled, headroom works.  When enabled, headroom breaks.
Now, I am only able to get headroom to work on codepen and not my actual server, even when I disable md-menu. :-( .
That means it might have something to do with the use of an angular ui-router and how the HTML is built using partials.  Have you heard of any such issue?
The dependencies for the codepen are:

1. Actual angular.js library installed locally on my server (https://pelotonstreet.com/js/libraries/angular.min.js)

2. Headroom.js from (https://npmcdn.com/headroom.js), which is the same one installed locally.

3. jQuery from (https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.2/jquery.min.js)

4. Angular-headroom.js from another codepen (https://codepen.io/frankcastro/pen/VaOOpp), which is identical to the one installed locally.

Am I missing a reference or something with the jquery library?

@norkunas After much trial-and-error, I managed to get it work.  Thanks so much for your help.

@tabish89 glad you resolved it :+1:

@tabish89 glad you got it resolved :) It might be worth explaining here how you fixed it, should anyone else have the problem in future...
Also, a word on etiquette (since you said you're a novice) - it's not cool to spam github issues with requests for help. You commented on 4 separate issues, which is totally unnecessary. For personal help requests (where there is clearly not a bug), you would be better served posting to StackOverflow where there is a veritable army of people happy to help diagnose problems.
And thanks @norkunas for stepping up to assist

@WickyNilliams You know you spend a lot of time telling people about etiquette, but perhaps you should spend that time updating the poor documentation that accompanies your library.  So call it what you want.
You disappear for months on end, but yet log on to tell someone about etiquette or how they're "being rude."  Frankly, just reading your comments and responses irk me - almost hypocritical (yes, I'm calling you out). You really think that you're God's gift to humanity?!?  I understand that it is open source code and that you developed headroom.js on your own time free of charge (tell me how big of a violin you want me to play), but you should still take ownership of your thing otherwise remove it from the web/domain all together.  I think there are several people here who would directionally agree with me.
I tried StackOverflow, CodePen, and a few others -- and most of the interaction on those sites were over 2 years old, so that tells me people have generally given up.  I did however notice some more recent comments on Github; and, decided to reach out to others in the community and see if we could solve the issue (because of your subpar and lacking documentation) together.  That's when @norkunas stepped up to plate and kindly pointed me in the right direction.

For the rest of the community, here is codepen to see how I fixed the issue: codepen
Couple of notes:

Take a look at the JS dependencies by clicking on the "gear" icon in the JS window.  You will notice that there is one external dependency to another codepen created by Frank Castro, which is essentially the anguar.headroom.js wrapper.  I used this external link instead of the one of our server, because all of the development related to headroom.js has been done on a branch that does not have an externally facing IP.  Our production server has not been synched up with our dev branches, since there is still a lot of work in progress.

Some salient differences in the CSS.  What is provided in the documentation would not work, but this seems to have worked.  Not sure why... but, it's above my pay grade.

In some cases, md-menu in Angular can cause some conflicts -- so, it is disabled in the codepen.

@tabish89 to be fair, GitHub is not the place to ask the questions you have been asking. No repository owner/maintainer should be forced to have to respond to everyone by a certain time - you have no right to make that demand on someone else’s time whatsoever. You also can fork anything put in the open if you so wish - nobody is stopping you from doing so.
Comments like that frankly are why a lot of good maintainers don’t want to work on open source software anymore. Offloading personal frustrations over using someone’s library to increase other people’s stress is not ok. Unwarranted disrespect hurts every user.
Filing an issue over insufficient documentation on how to do certain things would be ok - it is important to remember that the informal contract is that maintainers give their time to improve libraries, almost always out of charity, but there is an equal burden for users to be good actors & foster respectful collaboration.

@wesleycho Fair.  I see your point.  It's all in the delivery.  I had tried to be very careful in not demanding any feedback or response by a certain time -- and, apologize if my comments were misconstrued as such.

@tabish89 which exact parts of my comment elicited that response? It's absolutely baffling that you think that is a reasonable way to respond. I had, up until then, been nothing but polite and courteous. That may not continue. And worse yet, you seem proud of how you behave, given that trite "Mike drop" sign off (it's mic btw). It is nothing to be proud of, it is unequivocally shameful behaviour.
You are, by your own admission, a novice. Therefore you are likely inexperienced in dealing with open source projects. So I tried to explain (politely, I might add) some things for your benefit, and you come back with insults and an abrasive attitude.
So let me break it down, so that there is no room for further misunderstanding...

I am extremely appreciative of your goodwill. I'm a coding novice, so your patience is also much appreciated in advance.

Your goodwill, patience, and appreciation didn't last long, did it? If you're inexperienced, how about listening, and taking the advice of someone who is not?

You disappear for months on end

Because I have a life, and I can choose to work on my projects whenever I want.

yet log on to tell someone about etiquette

Because you obviously do not understand the etiquette, and it absolutely needs explaining. How you acted initially suggested this was the case, and how you followed up proved it beyond a shadow of a doubt.

You really think that you're God's gift to humanity?!?

Ummm... no? I merely expect to be spoken to, and treated with, basic decency. This relationship is not mutually beneficial.  Congratulations, I guess?). So how about being a little more cordial in future, seeing as it is you that is getting all the benefit out of this relationship.

I tried StackOverflow, CodePen, and a few others

But did you post to stack overflow? That's what I was suggesting is the best course for future. The reason being I know that 90% of issues raised here are not bugs but implementation issues. Or to be curt: it is your problem, not mine. All you do is distract from real issues by posting personal help requests here.

and most of the interaction on those sites were over 2 years old, so that tells me people have generally given up.

Or, the more likely scenario, that most people are not having issues and are able to use the library without issue. This library is used by a huge number of websites (>2k of the top 1m sites!), all without issue. Did you stop to consider that your code might be the problem?

I understand that it is open source code and that you developed headroom.js on your own time free of charge (tell me how big of a violin you want me to play), but you should still take ownership of your thing otherwise remove it from the web/domain all together.

I am under no such obligation. Nor is any other open source maintainer. Have you read the MIT license? It says as much: "This software is provided as is". You are free to fork this project, write a competing project from scratch, or make pull requests if something is lacking. I encourage you to do so!

I think there are several people here who would directionally agree with me.

I am struggling to imagine a single person agreeing with you. In fact, I expect people would be unanimously appalled by your attitude.

I did however notice some more recent comments on Github; and, decided to reach out to others in the community.

You did not reach out to the community, as this is not a community, nor is it a place to make personal help requests - it is a bug tracker. You spammed several issues here instead of replying to my offer of help on twitter (and continued to spam issues after I asked you not to via twitter). Even creating a brand new issue would have been preferable, since at least then only I'm getting notifications, rather than the 10 or so people in all the issues you commented on.

(because of your subpar and lacking documentation)

Where was it lacking, and in what way? I assume you're working on a PR to improve the docs? Right?

I see your point. It's all in the delivery. I had tried to be very careful in not demanding any feedback or response by a certain time -- and, apologize if my comments were misconstrued as such.

Nope, you've completely missed the point. The point is you're acting rude, insulting, and entitled. That is what you should apologise for, and what you should avoid in future.
TL;DR: I think it is you, sir, that should "check yourself", because you are acting obnoxious and entitled, which is not how you be a good member of the open source community. I hope your future interactions with maintainers will be less hostile, and wish you the best.

@WickyNilliams
You’re correct... I did not see your response to my Twitter request.  In my ripe old age of 42, I'm NOT all that plugged into Twitter and Instagram... but, that's not an excuse.
Given that outreach on your part, I am out of line.  Global apology (more, for wasting your time in having to write such a length response).  When I received your response this morning, I was under the (wrong) impression that you were accosting me (like you have to a few others here) from the get go - which is what incited my rude response back to you.  Like I said... out of line.
Lesson learned.

Thank you for being mature and polite in this response.
But! It is out of line whether I had replied on twitter or not. Read my original comment again. There is nothing there that should provoke that response, it started with "glad you got it resolved :)" - clearly friendly in tone. The twitter element has very little bearing.
I reject the notion I have accosted anyone. Accosting is a very strong word. People have brought attitude and a sense of entitlement here time and time again. It is not accosting someone to call it out. Where I have been wrong I have apologised (communicating in text can be difficult). However that attitude and rudeness is common in OSS. If you ever release a project that gains some success you will see it first hand. Speak to any maintainer and they will tell you the same. It is draining and demoralising. For the most part I now avoid open source because of it. Which is why, as you observed, I only drop in every couple of months, and only because I feel some guilt for neglecting my projects.

BTW, did you have any suggestions to improve the docs? Happy to update them if there's any glaring omissions

FWIW, at for the Angular implementation since it can be a bit of a crapshoot, a working codepen based that shows live/working example would be helpful.   You might consider cleaning up by forking what I used to troubleshoot the issue:


confirm pivot block - removed disconnect based on chain height estimate

PR description
When choosing the "best" peer to try to confirm the pivot block

use the default comparator (TTD then chain height) effectively the same
remove the pre-requirement for the chain height estimate
don't disconnect the "worst" useless peer at this point
and update the peer's chain height estimate if the pivot block is confirmed
also allow 5 peers below the pivot block height since the chain height estimate we use for comparison can be wildly inaccurate

Fixed Issue(s)
Fixes #6887
Thanks for sending a pull request! Have you done the following?

[ ] Checked out our contribution guidelines?
[ ] Considered documentation and added the doc-change-required label to this PR if updates are required.
[ ] Considered the changelog and included an update if required.
[ ] For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

[ ] unit tests: ./gradlew build
[ ] acceptance tests: ./gradlew acceptanceTest
[ ] integration tests: ./gradlew integrationTest
[ ] reference tests: ./gradlew ethereum:referenceTests:referenceTests

3 EthPeers {
PeerId: 0x3bd1127b27212d2d... 1297018 height (est) PeerReputation score: 97, timeouts: {1=4, 3=1}, useless: 0, validated? true, disconnected? false, client: erigon/v2.58.1-f12e451c/linux-amd64/go1.21.7, [Connection with hashCode 820003208 inboundInitiated? false initAt<PHONE_NUMBER>564], enode://3bd1127b27212d2d165ffaa7ae1e0efbdfdf7ff2022e851dc95271e44bb271446ab05e54035756da7eb6ce86b595992d94f2400ed19a34d9485df81fc12d0879@<IP_ADDRESS>:30304,
PeerId: 0xac906289e4b7f12d... 1296969 height (est) PeerReputation score: 150, timeouts: {}, useless: 0, validated? true, disconnected? false, client: Geth/v1.13.14-stable-2bd6bd01/linux-amd64/go1.21.7, [Connection with hashCode<PHONE_NUMBER> inboundInitiated? false initAt<PHONE_NUMBER>663], enode://ac906289e4b7f12df423d654c5a962b6ebe5b3a74cc9e06292a85221f9a64a6f1cfdd6b714ed6dacef51578f92b34c60ee91e9ede9c7f8fadc4d347326d95e2b@<IP_ADDRESS>:30303,
PeerId: 0xa3435a0155a3e837... 1297002 height (est) PeerReputation score: 150, timeouts: {}, useless: 0, validated? true, disconnected? false, client: Geth/v1.13.14-stable-2bd6bd01/linux-amd64/go1.21.7, [Connection with hashCode 455188871 inboundInitiated? false initAt<PHONE_NUMBER>663], enode://a3435a0155a3e837c02f5e7f5662a2f1fbc25b48e4dc232016e1c51b544cb5b4510ef633ea3278c0e970fa8ad8141e2d4d0f9f95456c537ff05fdf9b31c15072@<IP_ADDRESS>:30303}

2024-04-07 13:45:03.763
trailing peer requirements TrailingPeerRequirements{minimumHeightToBeUpToDate=1296943, maxTrailingPeers=5}

2024-04-07 13:45:03.763
trailing peer requirements TrailingPeerRequirements{minimumHeightToBeUpToDate=0, maxTrailingPeers=9223372036854775807}

I started 3 holesky nodes last night - 2 of 3 did really well, the third one dropped to zero peers (this seems to happen on holesky, workaround is to add the 2 bootnodes using admin_addPeer so they effectively become static peers). The first 2 still have 25 peers and the third one still has only 2 peers (the bootnodes) <- I have seen this on holesky independent of this PR (maybe worth investigating as a discovery issue separately but I don't think it's high priority)

holesky sync also fine (once the zero peers node got peers, it quickly got back in sync)

2 x mainnet nodes started today, one took a long time (more than 5h) to find peers - mainnet-sally-6889-07  - currently on 24 peers. Both look to be syncing fine (so far)

sync progress

this has been incorporated into #6968


  enable flex layout for content slots

Prerequisites

[X] I have read the Contributing Guidelines.
[X] I have not leaked any internal/restricted information like screenshots, videos, code snippets, links etc.

Suggestion / feature request
Implementing a responsive full-height flex layout in iX applications can be difficult.
Some components, such as ix-pane-layout, have a default display: block for their content slots.
Additionally, in some cases, these styles cannot be overridden due to the shadow root (e.g., ix-application > main.content).
This is the custom CSS I had to use to overcome this issue:
html, body {
  display: flex;
  width: 100%;
  height: 100%;
}

ix-pane-layout > [slot=content] {
  display: flex;
  flex: 1;
  width: 100%;
  height: 100%;
}

Hello @intuiquest!
Could you kindly provide us with a working stackplitz example, so we can analyse it in more detail?

Hello @intuiquest, for your use case I would suggest to wrap your slotted content in a div with display: flex.


[DOC] Add hacktoberfest as a topic for this repo

Hacktoberfest has made participation for maintainers to the event as opt-in
https://hacktoberfest.digitalocean.com/hacktoberfest-update
@gimseng To get contributions for hacktoberfest you would need to add a 'hacktoberfest' topic in the about section.
Refer to the above guide for more info.

Hi @AjayKhalsa thanks ! I just did that. I see that you/someone else added the appropriate label for PR. Thanks ! Feel free to close this when you are satisfied.


fix VerbosityZero check panic

try to fix https://github.com/kubernetes-sigs/logtools/pull/4#discussion_r958364030

I cannot reproduce the original problem anymore, so let's keep the code as it is.
/close

yeah, I agree.

/reopen


feature(argus.backend): Profile Jobs tab v2

This commit revises the way assigned jobs are presented to the user
on the profile jobs page. First, the jobs are split into "Investigated"
and "Not Investigated" categories, with succesfully passed jobs
automatically being added to "Investigated" ones. Second, the jobs are
filtered server-side to only apply to current schedule + 1 week behind.
Lastly, a new "filter by name" control was added to simplify looking for
specific jobs

Trello

Suggested edit:
diff --git a/argus/backend/argus_service.py b/argus/backend/argus_service.py
--- argus/backend/argus_service.py
+++ argus/backend/argus_service.py
@@ -1032,23 +1032,15 @@
         valid_runs = []
         for run in runs:
             run_date = datetime.datetime.fromtimestamp(run["start_time"])
             for schedule in schedules:
+                if not schedule["period_start"] < run_date < schedule["period_end"]:
+                    continue
                 if not run["release_name"] == schedule["release"]:
                     continue
-                if not schedule["period_start"] < run_date < schedule["period_end"]:
-                    continue
                 if run["assignee"] in schedule["assignees"]:
                     valid_runs.append(run)
-                    continue
-                group = run["group"].split("_")[0]
-                if group in schedule["groups"]:
-                    valid_runs.append(run)
                     break
-                filtered_tests = [test for test in schedule["tests"] if re.match(f"^{test}(-test)?$", run["name"])]
-                if len(filtered_tests) > 0:
-                    valid_runs.append(run)
-                    break
         return valid_runs

     def get_schedules_for_user(self, user: User):
         all_assigned_schedules = ArgusReleaseScheduleAssignee.filter(assignee=user.id).all()


Notify other maintainers on package claim

Reticketed from #4305.

This is complicated because our current infrastructure assumes that an email address is linked to a participant. Either we need to create stub participants for all relevant emails (we do something similar for social accounts, but then I guess we want to be able to "promote" the stubs? does that get us into #1052?) or undo that assumption so we can mail without having a participant.

+1 from @rohitpaulk at https://github.com/gratipay/gratipay.com/pull/4579#issuecomment-324401774:

The email body should clearly mention that the Gratipay user is a verified owner of that specific email address.


Fetch support in IE9+

Hello Mike!
Seems like fetch is not available in IE yet, but it will be a big deal in all d3 projects.
Do you have plans to provide a fallback for IE9+?
or should the browser support be updated to "no IE anymore"?

D3 supports “modern” browsers, which generally means everything except IE8 and older versions. D3 is tested against Firefox, Chrome, Safari, Opera, IE9+, Android and iOS.

No plans to support IE in D3 5+, but a Fetch polyfill would probably work for d3-fetch. I’ll update the wiki when 5.0.0 is released.


Remove note in datasetfilewriter about now-closed ticket

Description

I removed the note
note was about this: #19028

Type of change

[ ] Bug fix
[ ] New feature
[ ] Documentation update
[x] Other~~

How Has This Been Tested?

n/a
Reminders:

Please follow the style guidelines of this project.
Please perform a self-review of your code before submitting a PR and asking others to review it.
Please assign reviewers (see VisIt's PR procedures for more information).

Checklist:

[ ] I have commented my code where applicable.
[ ] I have updated the release notes.
[ ] I have made corresponding changes to the documentation.
[ ] I have added debugging support to my changes.
[ ] I have added tests that prove my fix is effective or that my feature works.
[ ] I have confirmed new and existing unit tests pass locally with my changes.
[ ] I have added new baselines for any new tests to the repo.
[x] I have NOT made any changes to protocol or public interfaces in an RC branch.~~

merge w/ develop in #19707


Adding isDirectory method to pinotFS

Adding isDirectory method so that we can tell whether the uri is a file or directory.
This method is needed to support deletion / retention

Codecov Report

Merging #3339 into master will increase coverage by 0.05%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master    #3339      +/-   ##
============================================
+ Coverage     70.69%   70.75%   +0.05%
  Complexity        4        4
============================================
  Files           985      985
  Lines         45424    45430       +6
  Branches       6098     6098
============================================
+ Hits          32114    32144      +30
+ Misses        11193    11174      -19
+ Partials       2117     2112       -5

Impacted Files
Coverage Δ
Complexity Δ

...in/java/com/linkedin/pinot/filesystem/PinotFS.java
100% <ø> (ø)
0 <0> (ø)
:arrow_down:

...a/com/linkedin/pinot/filesystem/HadoopPinotFS.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...va/com/linkedin/pinot/filesystem/AzurePinotFS.java
15.71% <0%> (-0.47%)
4 <0> (ø)

...va/com/linkedin/pinot/filesystem/LocalPinotFS.java
80% <0%> (-1.82%)
0 <0> (ø)

...in/pinot/core/query/pruner/ValidSegmentPruner.java
57.14% <0%> (-28.58%)
0% <0%> (ø)

...nsport/netty/PooledNettyClientResourceManager.java
85.41% <0%> (-4.17%)
0% <0%> (ø)

...inot/transport/netty/NettyTCPClientConnection.java
82.31% <0%> (-3.41%)
0% <0%> (ø)

...e/operator/dociditerators/MVScanDocIdIterator.java
60.6% <0%> (-3.04%)
0% <0%> (ø)

...linkedin/pinot/common/metrics/AbstractMetrics.java
77.37% <0%> (-1.46%)
0% <0%> (ø)

...core/startree/operator/StarTreeFilterOperator.java
95.75% <0%> (-0.61%)
0% <0%> (ø)

... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de24a02...656f4d1. Read the comment docs.


'DynamoDbContext.LoadAsync' does not respect 'DynamoDBOperationConfig.IndexName'

Description
When using the LoadAsync method on IDynamoDbContext to load an entity using a secondary global index, I'm getting unexpected behavior: an InvalidCastException is being thrown.
Switching the exact same call to a QueryAsync fixes the problem.
I believe LoadAsync is not respecting the IndexName property on the configuration object and is using the primary index for all searches. Either the API is incredibly misleading (as it allows you to pass this unused value) or there is a bug there.
Reproduction Steps

Create a table with a primary and a secondary index
Map the primary key to a Guid property
Map the secondary key to a string property

    [DynamoDBTable("Somethings")]
    internal sealed class Something
    {
        [DynamoDBHashKey]
        public Guid? Id { get; set; }

        [DynamoDBGlobalSecondaryIndexHashKey]
        public string? Name { get; set; }
    }

Try to use LoadAsync with a string key and pass new DynamoDBOperationConfig { IndexName = <Name_Of_Secondary_Index> } as options

await dynamoDBContext.LoadAsync<Something>("someName", new DynamoDBOperationConfig { IndexName = "Name-index" });

An InvalidCastException is thrown
Replace the LoadAsync call with QueryAsync, and it works as expected:

var query = dynamoDBContext.QueryAsync<Something>("someName", new DynamoDBOperationConfig { IndexName = "Name-index" });
await query.GetNextSetAsync();

Logs

System.InvalidCastException: Unable to cast object of type 'System.String' to type 'System.Guid'.
at Amazon.DynamoDBv2.Converter`1.TryTo(Object value, Primitive& p)
at Amazon.DynamoDBv2.Converter.TryToEntry(Object value, DynamoDBEntry& entry)
at Amazon.DynamoDBv2.Converter.ToEntry(Object value)
at Amazon.DynamoDBv2.DynamoDBEntryConversion.ConvertToEntry(Type inputType, Object value)
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ToDynamoDBEntry(SimplePropertyStorage propertyStorage, Object value, DynamoDBFlatConfig flatConfig, Boolean canReturnScalarInsteadOfList)
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ToDynamoDBEntry(SimplePropertyStorage propertyStorage, Object value, DynamoDBFlatConfig flatConfig)
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ValueToDynamoDBEntry(PropertyStorage propertyStorage, Object value, DynamoDBFlatConfig flatConfig)
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.MakeKey(Object hashKey, Object rangeKey, ItemStorageConfig storageConfig, DynamoDBFlatConfig flatConfig)
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.LoadHelperAsync[T](Object hashKey, Object rangeKey, DynamoDBOperationConfig operationConfig, CancellationToken cancellationToken)
at Amazon.DynamoDBv2.DataModel.DynamoDBContext.LoadAsync[T](Object hashKey, DynamoDBOperationConfig operationConfig, CancellationToken cancellationToken)

Environment

SDK Version: AWSSDK.Core Version="<IP_ADDRESS>"
Package Version: AWSSDK.DynamoDBv2 Version="<IP_ADDRESS>"
OS Info: Windows 10
Build Environment Visual Studio 2019 16.8.1
Targeted .NET Platform:  Net Core 3.1

Resolution

[ ] :wave: I can/would-like-to implement a fix for this problem myself

I belive LoadAsync should be changed to respect the index being passed as a parameter and use the object hashKey against that.

This is a :bug: bug-report

Hi @julealgon,
Good afternoon.
Please refer to the page .NET: Object Persistence Model -> Supported Data Types for the list of supported data types. Looks like Guid is not the supported data type. Refer to page Mapping Arbitrary Data with DynamoDB Using the AWS SDK for .NET Object Persistence Model for example on how support arbitrary data types. Basically it involves developing custom TypeCpnverter.
As a workaround, you may user string instead of Guid type for Id property. For using Guid type, modifying your code to below works fine:
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.DynamoDBv2.DocumentModel;
using System;

namespace DynamoDbSecondaryIndexTest1748
{
    class Program
    {
        static void Main(string[] args)
        {
            AmazonDynamoDBClient amazonDynamoDBClient = new AmazonDynamoDBClient();
            DynamoDBContext dynamoDBContext = new DynamoDBContext(amazonDynamoDBClient);

            var loadResult = dynamoDBContext.LoadAsync<Something>("75f05566-2209-4f3d-ba5a-cfd9da5eab62", new DynamoDBOperationConfig { IndexName = "Name-index" }).GetAwaiter().GetResult();

            Console.WriteLine("Loaded Result: " + loadResult);
        }
    }

    [DynamoDBTable("TestTable")]
    internal sealed class Something
    {
        [DynamoDBHashKey]
        [DynamoDBProperty(typeof(GuidTypeConverter))]
        public Guid Id { get; set; }

        [DynamoDBGlobalSecondaryIndexHashKey]
        public string Name { get; set; }

        public override string ToString()
        {
            return $"[Id: {Id}, Name: {Name}]";
        }
    }

    public class GuidTypeConverter : IPropertyConverter
    {
        public DynamoDBEntry ToEntry(object value)
        {
            Guid guidValue;
            string stringValue = value as string;
            if (value == null || !Guid.TryParse(stringValue, out guidValue)) throw new ArgumentOutOfRangeException();

            DynamoDBEntry entry = new Primitive
            {
                Value = guidValue.ToString()
            };

            return entry;
        }

        public object FromEntry(DynamoDBEntry entry)
        {
            Primitive primitive = entry as Primitive;
            Guid guidValue;
            if (primitive == null || !(primitive.Value is String) || string.IsNullOrEmpty((string)primitive.Value) || !Guid.TryParse((string)primitive.Value, out guidValue))
                throw new ArgumentOutOfRangeException();

            return guidValue;
        }
    }
}

NOTE: The above code should be used only for reference. Kindly validate and test the GuidTypeConverter logic before trying to use the Guid type converter. We are not responsible for any data loss or production issues.
Please let me know if this helps and if this issue could be closed.
Thanks,
Ashish

@ashishdhingra I believe you might have misunderstood my request here. Please note that Guid is working fine when loading and saving entities. I have both the primary key, as well as another property on my model, mapped as Guid.
The problem here happens with the global secondary index, which has a string mapped property. Once I try to use LoadAsync passing a string value and the name of the secondary index, I get the exception above which is attempting to convert my string argument into Guid (which does not make sense since I'm not querying on the primary index).
I kindly request that you take another look at this. Even if the page you provided doesn't mention Guid as a supported type, I believe the issue here is not directly related to that.

Also @ashishdhingra , I'd suggest to avoid providing example source code where you block on async code, as that is a bad practice that can lead to hard to debug problems in .Net. In your sample Main, you could just make Main itself async.
People not aware of the issues might end up naively copying those code sections into their code.

Hi @julealgon,
Good morning.
Thanks for your inputs. I used your code exactly where it tries to use the secondary index, to reproduce the issue. When not using the type converter, the QueryAsync call didn't throw any exception, but it also didn't returned any result. The LoadAsync call threw the type cast exception mentioned. After the use of type converter, the LoadAsync call (with the secondary index as 2nd parameter), worked properly. I will try to pinpoint the exact code in SDK which throws the error.
In the mean while, in order to get proper understanding of the issue, please share the following:

Exact DynamoDB column configuration in AWS Console and the sample code which is working fine for you when loading and saving entities, without use of a type converter.

Reconfirm what you meant by The problem here happens with the global secondary index, which has a string mapped property. Once I try to use LoadAsync passing a string value and the name of the secondary index,. I guess you are referring to the code below (which is what I used exactly):

dynamoDBContext.LoadAsync<Something>("75f05566-2209-4f3d-ba5a-cfd9da5eab62", new DynamoDBOperationConfig { IndexName = "Name-index" })

Regarding the example code where I'm blocking async call, that is just for reference purposes as mentioned in the note. I agree with your assertion that we should not block async calls in live production code.
Thanks,
Ashish

@ashishdhingra , can you clarify on this?

After the use of type converter, the LoadAsync call (with the secondary index as 2nd parameter), worked properly.

Do you mean you can use LoadAsync with the secondary index and actually return the data, or is just not returning anything? If it is not returning anything even though there is data with that value in the secondary index, then it confirms my suspicion that it is actually ignoring the IndexName setting and actually searching on the primary index instead.

In the mean while, in order to get proper understanding of the issue, please share the following:

Exact DynamoDB column configuration in AWS Console and the sample code which is working fine for you when loading and saving entities, without use of a type converter.

Primary index:

Partition key: Id (string)
Sort key: -

Global Secondary index:

Name: Name-index
Partition key: Name (string)
Sort key: -
Attributes: Id, Name

This configuration allows me to:

Use LoadAsync with the primary key (.Net Guid): objects are fetched properly
Use Save with my entity: entity is persisted properly, including Guid ID which I generate before the call
Use QueryAsync().GetNextSetAsync().SingleOrDefault() with the secondary key + index name to fetch the entity using the secondary index

I cannot currently use LoadAsync to fecth from the secondary index.

Reconfirm what you meant by The problem here happens with the global secondary index, which has a string mapped property. Once I try to use LoadAsync passing a string value and the name of the secondary index,. I guess you are referring to the code below (which is what I used exactly):

Affirmative. That call produces InvalidCastException which I believe is incorrect behavior: it should not be attempting to cast the string into a Guid since the secondary index property that is marked in my model is not a Guid, but a string property:
   ...
        [DynamoDBHashKey]
        public Guid Id { get; set; }

        [DynamoDBGlobalSecondaryIndexHashKey]
        public string? Name { get; set; }
   ...

nor should it require the creation of a converter from string to Guid as Guid works out of the box.

I performed another test on my end which I figured could potentially fix the issue.
I realized that the DynamoDBGlobalSecondaryIndexHashKey attribute has an index name property, so I changed my model to:
        [DynamoDBHashKey]
        public Guid Id { get; set; }

        [DynamoDBGlobalSecondaryIndexHashKey("Name-index")]
        public string? Name { get; set; }

However, I still get the same error when attempting to use LoadAsync.

Hi @julealgon,
Upon further investigating and using the code below (NOTE: To keep things simple, I used .GetAwaiter().GetResult()):
using Amazon.DynamoDBv2;
using Amazon.DynamoDBv2.DataModel;
using Amazon.DynamoDBv2.DocumentModel;
using System;

namespace DynamoDbSecondaryIndexTest1748
{
    class Program
    {
        static void Main(string[] args)
        {
            AmazonDynamoDBClient amazonDynamoDBClient = new AmazonDynamoDBClient();
            DynamoDBContext dynamoDBContext = new DynamoDBContext(amazonDynamoDBClient);

            var loadResultSecondary = dynamoDBContext.LoadAsync<Something>("87a8f81a-9fe9-48cb-a7c0-a7a6788d91d3", new DynamoDBOperationConfig { IndexName = "Name-index" }).GetAwaiter().GetResult();
            Console.WriteLine("Loaded Result: " + loadResultSecondary);
        }
    }

    [DynamoDBTable("TestTable")]
    internal sealed class Something
    {
        [DynamoDBHashKey]
        public Guid? Id { get; set; }

        [DynamoDBGlobalSecondaryIndexHashKey]
        public string Name { get; set; }

        public override string ToString()
        {
            return $"[Id: {Id}, Name: {Name}]";
        }
    }
}

it gave the mentioned error, whether I used dynamoDBContext.LoadAsync<Something>("<some_guid_string_value>", new DynamoDBOperationConfig { IndexName = "Name-index" })or dynamoDBContext.LoadAsync<Something>("<some_guid_string_value>"). Using the below stack trace:
System.InvalidCastException
  HResult=0x80004002
  Message=Unable to cast object of type 'System.String' to type 'System.Guid'.
  Source=AWSSDK.DynamoDBv2
  StackTrace:
   at Amazon.DynamoDBv2.Converter`1.TryTo(Object value, Primitive& p)
   at Amazon.DynamoDBv2.Converter.TryToEntry(Object value, DynamoDBEntry& entry)
   at Amazon.DynamoDBv2.Converter.ToEntry(Object value)
   at Amazon.DynamoDBv2.DynamoDBEntryConversion.ConvertToEntry(Type inputType, Object value)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ToDynamoDBEntry(SimplePropertyStorage propertyStorage, Object value, DynamoDBFlatConfig flatConfig, Boolean canReturnScalarInsteadOfList)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ToDynamoDBEntry(SimplePropertyStorage propertyStorage, Object value, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.ValueToDynamoDBEntry(PropertyStorage propertyStorage, Object value, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.MakeKey(Object hashKey, Object rangeKey, ItemStorageConfig storageConfig, DynamoDBFlatConfig flatConfig)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.LoadHelperAsync[T](Object hashKey, Object rangeKey, DynamoDBOperationConfig operationConfig, CancellationToken cancellationToken)
   at Amazon.DynamoDBv2.DataModel.DynamoDBContext.LoadAsync[T](Object hashKey, DynamoDBOperationConfig operationConfig, CancellationToken cancellationToken)
   at DynamoDbSecondaryIndexTest1748.Program.Main(String[] args) in C:\Users\ashdhin\source\repos\DynamoDbSecondaryIndexTest1748\DynamoDbSecondaryIndexTest1748\Program.cs:line 19

I was able to trace the call to TryTo. Basically when you pass string value, it tries to use (Guid)<string_value> to get the Guid value type. This call fails giving error Cannot convert type 'string' to 'System.Guid'. You may try to validate this by executing (Guid)"87a8f81a-9fe9-48cb-a7c0-a7a6788d91d3" in Visual Studio immediate window, you will get the same error.
The workaround here is to execute the call dynamoDBContext.LoadAsync<Something>(Guid.Parse("87a8f81a-9fe9-48cb-a7c0-a7a6788d91d3"), new DynamoDBOperationConfig { IndexName = "Name-index" }) instead. Notice the use of Guid.Parse(). This works successfully without using type converter.
Please let me know if this works for you and let me know if this issue could be closed.
Thanks,
Ashish

@ashishdhingra

Basically when you pass string value, it tries to use (Guid)<string_value> to get the Guid value type.

Why though? It shouldn't be trying to do that, since the index I'm querying does not have a Guid as the hashkey: it is a string.

The workaround here is to execute the call dynamoDBContext.LoadAsync(Guid.Parse("87a8f81a-9fe9-48cb-a7c0-a7a6788d91d3"), new DynamoDBOperationConfig { IndexName = "Name-index" }) instead. Notice the use of Guid.Parse(). This works successfully without using type converter.

This is not a workaround, this is a different query altogether. Please note that the string Name property which is the hash key for the secondary index, is not a guid: it is actually textual data entered into the system by users, so it is mapped as string both on Dynamo as well as on my C# model class. This is contrary to the primary index hash key, which is a Guid and is mapped in Dynamo as string, and in code as Guid. The primary index Load is working fine, we are talking about the secondary index here.
I hope that clarifies the issue. Let me know if you need additional info.

Hi @julealgon,
The LoadAsync is part of DynamoDB Object Persistence model (similar to ORM frameworks for relational databases). For the mentioned problem, you have used Guild at C# object level instead of string at DynamoDb level. DynamoDB Object Persistence model can map arbitrary data types to DynamoDB properties via type converters. Also notice that first parameter of LoadAsync is of type object. Hence before executing the query, it tries to map the object type value to proper object (Something) property type (Guid) value. The cast (T)<some_value> is a valid call since DynamoDb out of box only supports simple types as mentioned earlier. Hope this gives some insights into 1st question. Feel feel to explore SDK code to find more details of Object Persistence design.
For the 2nd question, DynamoDB SDK's LoadAsync method calls GetItem service API operation. If you refer Service API reference for GetItem, it expects Key request parameter which is a A map of attribute names to AttributeValue objects, representing the primary key of the item to retrieve.. Hence in you case, it should be the Guid object property. The .NET API reference for DynamoDBContext.LoadAsync also specifies the first parameter as Hash key element of the target item.. I'm not sure on how using 2nd parameter with secondary index name affects the GetItem service operation.
I'm not sure if for your use case (per your 2nd question), you intend to use Query operation.
Hope this helps.
Thanks,
Ashish

The LoadAsync is part of DynamoDB Object Persistence model (similar to ORM frameworks for relational databases). For the mentioned problem, you have used Guild at C# object level instead of string at DynamoDb level.

Incorrect. The problem I'm having is not with the primary key (which is mapped as Guid in C# code) but with the hash key for the secondary index, which is string on both ends.

DynamoDB Object Persistence model can map arbitrary data types to DynamoDB properties via type converters.

I understand that, however the Guid primary key is working fine for me without having to create any converters. I think you should probably update your documentation to clarify the behavior around Guids in the Object Persistence Model, since they appear supported even though they are not listed there.

Also notice that first parameter of LoadAsync is of type object. Hence before executing the query, it tries to map the object type value to proper object (Something) property type (Guid) value.

I understand the error, but I don't understand why it is attempting to perform that conversion in the first place. The type it should be attempting to cast to for the secondary index is string, not Guid: I'm not performing an operation on the primary index here when specifying IndexName.

For the 2nd question, DynamoDB SDK's LoadAsync method calls GetItem service API operation. If you refer Service API reference for GetItem, it expects Key request parameter which is a A map of attribute names to AttributeValue objects, representing the primary key of the item to retrieve.

Now this is where it gets incredibly misleading then. The LoadAsync API names the first parameter hashKey, not primaryKey. Following that nomenclature, one would understand it is "the hashkey for the index being searched", since:

you can specify the index name as an additional parameter
every index in DynamoDB has its own hashKey

If it truly is referring to the primary  hashkey, then the API is incredibly misleading an needs work, in my opinion. Especially when directly compared to "sibling" APIs like QueryAsync, which takes the same parameters, but actually works on the index you specify and not only on the primary index.

I'm not sure if for your use case (per your 2nd question), you intend to use Query operation.

From the API surface, it would appear that LoadAsync should work on my scenario, but it doesn't. As I mentioned in the OP, I think either the API should be changed to support the scenario, or it should prohibit you from passing an index name into the method. The former is preferred in my view for API consistency (working similarly to QueryAsync).

Hi @julealgon,
With all above inputs I will work with developer to have a look at it, even though it might be a documentation update. I will add bug label back to get the traction.
Thanks,
Ashish

...even though it might be a .NET API documentation update.

Again, while changing the documentation to make this behavior explicit is a valid "fix", I personally think it is a bad option in this case as the APIs will "appear" to behave the same but work differently at the end of the day, and this is bad for consumers. Obviously, this is up to you, just wanted to make my position clear here.

Kindly notice that the GetItem service API call returns single item, as opposed to Query service API call which returns multiple items. Hence GetItem should really work on primary key which is unique for all items. Please validate if this explains the behaviour of LoadAsync.

Perhaps we are confusing "primary/secondary" with "hash/sort" keys here? My understanding was that secondary indexes also  had unique items under their respective hash keys, so the approach shouldn't change regardless if we are talking about single-result Load vs multi-result Query: using Load with a hash key from a secondary index should have similar behavior than performing the same call using the hash key for a primary index, i.e. it should always have "at most 1 result" in both cases.
Please do correct me if I'm wrong here as I just started interfacing with Dynamo.

API documentation for DynamoDBContext.Load (object) call which states Loads an object from DynamoDB for the given hash-and-range primary key and using the given config..

I never noticed the XML docs for the non-async call as I was always using the async version by default. This documentation does seem to suggest it only works on the primary index (having the primary key mentioned there), which would then result in a very misleading API as I said earlier since you can pass any other index name that is not the primary one as part of the call.
If the documentation is really correct here and this is not a bug, then I would strongly recommend removing the capability of passing the index name in the Load/LoadAsync calls. At least that will move developers to the pit of success faster.

...I will work with developer to have a look at it,...
I will add bug label back to get the traction.

I appreciate your support on this Ashish.

@julealgon Adding more context here. As @ashishdhingra mentioned LoadAsync is basically a wrapper around the underlying GetItem Service API. Basically it calls that and takes care of converting the low level service client response into the POCO.  All of the operations on the DynamoDBContext have an override that take in the same DynamoDBOperationConfig type. I admit not my favorite design as it caused DynamoDBOperationConfig to be a union of all possible config settings. That means some settings like IndexName won't always be applicable. At this point changing this would be a serious breaking change which we are not ready to make.
I agree the parameter name on the LoadAsync  would be clearer if it was called primary key. For background when this code was first written DynamoDB didn't have indexes yet and this just wasn't something we thought about going back and renaming.
As for the idea of making LoadAsync work like QueryAsync except for a single object I think that could also cause serious confusion. A hash and range key for an index is not guaranteed to be unique in DynamoDB. So if we got multiple items back we would have to either throw an error which could be very unexpected or return back the first item which would be unpredictable.
At this point the best we can do is improve the documentation. Anything else causes serious breaking changes.

Thank you @normj for the information.

At this point changing this would be a serious breaking change which we are not ready to make.
At this point the best we can do is improve the documentation. Anything else causes serious breaking changes.

If you are not willing to change the behavior of the API, perhaps you could introduce new overloads, with a tighter config model (containing only the settings that make sense for any given operation) while retaining the old overload as-is, but marking it as deprecated. That would allow you to slowly phase it out and replace with the newer, more robust one.
Another potential option would be to actually throw an exception at runtime when an invalid setting is passed into the call. I'd rather get an exception that tells me why that setting cannot be used, than get "wrong results" as is happening right now.
Anyways, very unfortunate to have such a misleading API stuck in place. I'll probably need a significant comment block in my code to explain this anomaly and justify using the Query syntax in my repository implementation. I hope you can look back at this at some point and rethink the strategy (maybe implement an EF provider instead of a custom model, like CosmosDB does and some paid providers provide). Clearly, the "wheel reimplementation" here created unnecessary problems for consumers.

Just spent a day banging my head into the wall trying to LoadAsync() from a GSI having only a hash (but no sort) key which means there must be exactly one result matching the key. Having to use QueryAsync().GetRemainingAsync().Single() in such case is not very intuitive.

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.

From my perspective, this is still valid, even if its a documentation-only change.

Needs review on how we could implement this documentation change.

Wow, I just spent hours trying to work out what I was doing wrong! This really needs to be included in the documentation if it's not going to be fixed. It would've saved me so much headache!

For the short term, we've clarified the Load/LoadAsync documentation starting in DynamoDBv2 v3.7.303.9 via #3320 to explain that 1) it calls GetItem under the hood, which relies on the primary key and 2) IndexName on the config does not influence which item is loaded.
Longer term, we would like to make the change @julealgon suggested above - move away from the "shared" DynamoDBOperationConfig towards operation-specific configs that only contain applicable settings for each. You may have noticed the v4-development branch, this is on our list of desired DynamoDB changes along with the interface/mockability issues.

We've merged #3421 into the v4-development branch, which includes a new set of operation-specific config objects LoadConfig, SaveConfig, QueryConfig, etc. This should help avoid confusion about IndexName and other properties that only apply to a subset of operations moving forward.
We've marked the methods that take DynamoDBOperationConfig as deprecated in V4, but are not planning to remove them entirely yet. We may do so in a future major version.
We're still working on getting preview packages published, you can track overall V4 progress in #3362. We'll leave this issue open until at least V4 hits GA.

The V4 preview is out, please let us know if you find any issues with the new operation-specific config objects.


The Cosign integration doesn't work with version 2.0.0+

Before version 2.0.0, Cosign would generate private keys with the following labels:
-----BEGIN ENCRYPTED COSIGN PRIVATE KEY-----
...
-----END ENCRYPTED COSIGN PRIVATE KEY-----

Since version 2.0.0, the labels used for the private keys have changed to the following:
-----BEGIN ENCRYPTED SIGSTORE PRIVATE KEY-----
...
-----END ENCRYPTED SIGSTORE PRIVATE KEY-----

The changes have been made in https://github.com/sigstore/cosign/issues/2471.
That means the cosign generate-key-pair command (used in the tutorial) doesn't work anymore because kpack can't read the new format until the cosign dependency is upgraded in kpack.
I'm available to contribute a pull request, if needed.

I tried to give the change a try in https://github.com/pivotal/kpack/pull/1197