Retrieval of Chart List Results in Failure in Large Clusters

Description
Hi, thanks for this project, I can see alot of value in it!
Our organisation has close to 1000 namespaces in some clusters, and the following command fails retrieving the list of charts:
level=warning msg="Failed command: [helm ls --all --output json --time-format 2006-01-02T15:04:05Z07:00 --all-namespaces]"
We suspect the list of items it has to return results in a timeout or similar.
Thanks again!
Screenshots

Additional information
No response

Can you check the execution time for the same command from cli and see how long does it take?
time helm ls --all --output json --time-format 2006-01-02T15:04:05Z07:00 --all-namespaces

Hi @hdm23061993, I just ran this locally against the cluster I was trying to utilise helm-dahsboard with, and the timing results are as follows:
Executed in   65.79 secs    fish           external
   usr time   29.01 secs    0.09 millis   29.01 secs
   sys time    5.25 secs    2.04 millis    5.25 secs```

Upon further investigation, turns out that the helm ls command was being killed. It was a bit confusing as the container wasn't OOMing, and the pods weren't restarting though.
However, upon viewing the kernel logs on the worker nodes, we noticed that indeed the cgroup memory limits were being exceeded and the helm processes was being reaped.
We increased memory limits for the dashboard by a fair amount, and this has fixed the issue.
TL;DR
Pod needed much more memory allocated.
Happy to close if everyone else is.

Yup, the issue can be closed
Thanks 😊

@undera @hdm23061993
Just a follow up; I still think there may be a retrival time issue. We still get that red popup "Failed to get list of charts". But this time, the process is still not killed (now that it has more memory). I then hit refresh on the browser, and the list of releases/namespaces shows up.
Again, might or may not be related to the time it takes to receive the helm ls result. Worth an issue? More of a UI improvement.

I should say the scale you mention is quite far from what I had in mind while creating the helm-dashboard :) We'll need to rethink quite a bit the way we retrieve and present data in such cases

Pagination is the way to go in my opinion


Add KMAC support to KBKDF.

Now that ACVP test vectors exist, support has been added for this mode. See https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-108r1.pdf
Note that the test vectors used fairly large values for the input key and the context, so the contraints for these has been increased from 256 to 512 bytes.

Checklist

[x] documentation is added or updated
[x] tests are added or updated

Great work!

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Merged to master and 3.1 branches. Thank you.


NIFI-8098 - Bump jetty.version from 9.4.34.v20201102 to 9.4.35.v20201120

dependabot[bot]<EMAIL_ADDRESS>Updates jetty-server from 9.4.34.v20201102 to 9.4.35.v20201120

Release notes
Commits

Thank you for submitting a contribution to Apache NiFi.
Please provide a short description of the PR here:
Description of PR
Enables X functionality; fixes bug NIFI-YYYY.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[x] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[x] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically main)?

[x] Is your initial contribution a single, squashed commit? Additional commits in response to PR reviewer feedback should be made on this branch and pushed to allow change tracking. Do not squash or use --force when pushing to allow for clean monitoring of changes.

For code changes:

[x] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[x] Have you verified that the full build is successful on JDK 8?
[x] Have you verified that the full build is successful on JDK 11?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
[ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI for build issues and submit an update to your PR as soon as possible.

Tested upgraded Jetty in a secure cluster. Checked the other web contexts work as expected. Tested out using ListenHTTP, PostHTTP processors, and used Site-To-Site to transmit data to an input port on the same cluster.

Merged to main, thanks @thenatog


Default options style change

Backbone is changing their styles from
options = options || {};

to
options = _.extend({}, options);

This newer style is safer for contributors, particularly when addressing defaults so that you get
options = _.extend({ foo: 'bar' }, options);

vs
options = options || {};
if(!options.foo) options.foo = 'bar';

particularly to make sure defaults go at the top at not buried in the function somewhere.  Ref# https://github.com/jashkenas/backbone/pull/3629/files#diff-0d56d0d310de7ff18b3cef9c2f8f75dcL613

:+1: I like this


Fix issue #1406: Update lib.rs to see the expected output when deploying the Increment contract

What
Fix issue #1406: Update lib.rs to see the expected output when deploying the Increment contract, following this tutorial "https://developers.stellar.org/docs/build/smart-contracts/getting-started/deploy-increment-contract".
Why
I encountered an error when I run this command ('id' and 'source' were appropriately replaced).

Known limitations
[TODO or N/A]

@yskwmt is this the increment contract from the soroban-examples repo? I think the fix probably needs reopening there.


Fix a bug in SELECT DISTINCT ORDER BY.

Currently if the query is SELECT DISTINCT column1 FROM testTable ORDER BY column1 NULLS LAST, we get an error: ORDER-BY columns should be included in the DISTINCT columns.
Tested in unit tests and local server.

Codecov Report

Merging #10827 (c3e898a) into master (ab4e8fa) will decrease coverage by 54.77%.
The diff coverage is 0.00%.

@@              Coverage Diff              @@
##             master   #10827       +/-   ##
=============================================
- Coverage     68.41%   13.64%   -54.77%
+ Complexity     6518      439     -6079
=============================================
  Files          2164     2110       -54
  Lines        116410   113918     -2492
  Branches      17609    17311      -298
=============================================
- Hits          79637    15549    -64088
- Misses        31136    97097    +65961
+ Partials       5637     1272     -4365

Flag
Coverage Δ

integration1
?

unittests1
?

unittests2
13.64% <0.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...org/apache/pinot/sql/parsers/CalciteSqlParser.java
0.00% <0.00%> (-84.93%)
:arrow_down:

...uest/context/utils/QueryContextConverterUtils.java
0.00% <0.00%> (-97.85%)
:arrow_down:

... and 1670 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


hide gl controls behind a cogwheel icon

closes https://github.com/plotly/react-chart-editor/issues/636

yup, looks good 💃
after small comment


Naked returns outside top-level functions are ignored

None of the following naked returns are currently flagged. (For comparison, they are flagged by revive.)
var _ = func() (err error) {
	return
}

func foo() {
	var _ = func() (err error) {
		return
	}

	func() (err error) {
		return
	}()

	defer func() (err error) {
		return
	}()

	go func() (err error) {
		return
	}()
}

@alexkohler would you mind walking me through this problem? I would like to take this up or at least attempt it

hi @dibrinsofor! I'm not exactly what sure needs to be done to support these additional cases, but I imagine we'd just need to further tease apart the AST to identify these additional cases. We currently walk the AST here:
https://github.com/alexkohler/nakedret/blob/b2c01465241327a7db35ec77f94ff19168bcabaa/nakedret.go#L196-L235
I imagine we'd need to add some additional cases and potentially some additional logic within the existing cases. You should be able to add these cases into the testdata/ directory and use the tests under nakedret_test.go to validate your changes.

It looks like nakedret already identifies these cases, I would just like to extend the unit tests to include them.
would you mind hopping on a call to answer some questions? I want to rewrite the tests but just want to run somethings by you.


bug: Missing navbar on iOS PLUS devices

Type: bug
Platform: ios 8 browser
I have just checked out the latest nightly build  v1.1.0-nightly-1583 (cce78aa0cc) and noticed that the navigation bar is completely missing on PLUS devices. The issue does not affect regular devices (6, 6S).
The problem is reproducible using the default template:
ionic start myApp
cd myApp
bower install driftyco/ionic-bower#master
ionic build ios
open platforms/ios/myApp.xcodeproj
=> Emulate on 6 PLUS or 6S PLUS

Hey @Duffycola, does using the nightly build help?
http://code.ionicframework.com/nightly/js/ionic.bundle.js

Hi Perry, thanks for the reply.
Nope, I tested it using the v1.1.0-nightly-1583 build. It appears that the "nightly" build is 29 days old (see also https://github.com/driftyco/ionic-bower) ;-)
If there's a different location where I could get a bundle of a more recent version (say master branch or ios9-patch branch) I could give you more feedback.
Cheers,
Ed

Gave this a try on ios 6 plus and 6S plus using the latest bits, all looked well. The only thing that could be an the issue is the default zoom on the ios simulator. @Duffycola could you try pressing cmd-3 when the simulator is running?

@mhartington: Oh yeah, I pressed cmd-3 and cmd-2 and everything was fine. I then scaled back to original and realised that you can actually scroll up and see the status bar. So indeed it is just the fact that the iOS plus display height is larger than the available screen of my retina macbook and thus the simulator gets scaled down. Sorry for the confusion. How lame!

All good :+1:
Closing


Checking in for recurring events is slow

Report from Tramore dojo that when they have many dates it takes a long time to check someone in.
They have dates set up of 16th January til 12th March, 3 pages of attendees on one session, and 5 sessions.
Loading the dates for check in seems to be slow and could be improved.

I think I'll need a bit more info to narrow it down.
Is this then 20+ events, each having 5 sessions of regularly 1 page attending ppl?
Do we agree this is related to the action to check-in or to the action of displaying the list of users/sessions (and then not related to the event whatsoever)?

It seems to be a lib related issue(https://github.com/dotansimha/angularjs-dropdown-multiselect/issues/45) (no extra call done on the API side), trying to find a workaround as the lib sound not really maintained

Perfectly fixed now, thanks @Wardormeur ! :)


Docker updates

Re-opening this so we can move hosting forward. I have seen a few users having trouble hosting, and had quite a time getting it set up myself the first time. These changes represent a fairly standard way of web hosting and I think they are very beneficial to new users.
As I mentioned in the last PR the port change on the clients is intentional in order to use standard web ports (443 in prod) for hosting. If you are having issues with your hosting I would be glad to go through it and work out changes to support it.
I have been hosting using this docker setup for weeks now and have had zero issue. Please update with what issues you are running into so we can get this merged and out to the community.

I tested on my mac and found some problem:

this pr remove the js and dart 8443 and use caddy to proxy /ws to biz:8443
so i think must use caddy in scripts/webStart.sh

I download caddy by brew install caddy, but this can only download the 1.0.5
Then i download 2.0RC.3 from https://github.com/caddyserver/caddy/releases
I prefer brew because it's easy to use in scripts/*.sh

when i run 2.0, got some error

./caddy run -config docker/Caddyfile   22:23:41
2020/05/03 14:27:31.077	INFO	using provided configuration	{"config_file": "docker/Caddyfile", "config_adapter": ""}

Have you tried running this using only the docker-compose setup and the instructions listed in this branch?
For external or production hosting the node server should not be started. All js builds and proxying is handled by the web container, so all that is needed is to configure docker-compose and bring the stack up.
For local hosting I have been using the local.Caddyfile and the web container to provide js serving and biz socket proxy. This is similar to the prod config except SSL is not enabled and the exposed port is 8080 instead of 80 and 443.
For you #3 point I get a similar info message when booting my server, but everything works correctly.
{"level":"info","ts":1588521619.5022938,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}

Can you give it a shot just using the docker-compose setup and web container without the webStart.sh script.

I tested local hosting, it worked

uncomment build lines in docker-compose.yml
rm node_modules
docker-compose build
docker-compose up

I develop go code with scripts, diffcult to devleop in docker
I want to make scripts work, maybe i should ask @cloudwebrtc how to make 8443 configalbe

By the way:
Im doing this:
https://github.com/pion/ion/issues/157

With the new js sdk, the ws endpoint is now configurable

Going to merge this for now since we are moving the client apps and sdk out of the Ion repo. We will need to revisit front end hosting for the JS demo app when its final location is settled.

ok
maybe we need a new repo ion-web-app？or ion-web？


Add About CodeIN Community

Add goals , aim, showcase of our work and social media links to join our community

I can do this


Add support for semi required parameters

We have a dto
class AuthenticationDto {
    public String email;
    public String password;

    public String socialNetwork;
    public String accessToken;
}

And we have 2 ways to autheticate
POST /sessions
{"email"<EMAIL_ADDRESS>"password":"12345678"}

POST /sessions
{"social_network": "facebok", "access_token":"qqqwwweeerrtt"}

At the moment all fields are optional, but this not reflect that, if email is provided then password is required.
Proposal is to make password semi-required, this will indicate that is required in context, something like
AuthenticationDto {
    email (string, optional),
    password (string, semi-required{email}),
    social_network (string, optional),
    access_token (string, semi-required{social_network})
}

Or even better Swagger can show 2 models
AuthenticationDto {
    email (string),
    password (string)
}

AuthenticationDto {
    social_network (string),
    access_token (string)
}

The spec itself doesn't allow interdependent requirements, so we can't really implement it. The next version of the spec should have some support for it, but until then it can't be done.

Where can I find information about interdependent requirements,?

https://github.com/OAI/OpenAPI-Specification/issues/256


Dart fix expect files aren't being analyzed and contain errors

While writing Dart fixes, I noticed that I couldn't get the analyzer (flutter analyze ...) to properly report errors for the .expect files. Renaming material.dart.expect to material-expect.dart and running the analyzer revealed 3768 errors.
e.g. creating a ThemeData but not specifying all properties gives you a bunch of errors like:
  error • The named parameter 'switchTheme' is required, but there's no corresponding argument • material-expect.dart:945:15 • missing_required_argument

We should fix the errors that matter and probably use .dart for all .expect files. cc @Piinks, @bwilkerson any thoughts?

There is no analysis for the expect files because they are expected to have not fully functional code based on the transformation that has been applied.
For example, a transformation may add a parameter newParameter:  that is required, but left blank for the user to supply to value.
The expect files test that the transform is executed as expected and leaves it as simple as possible for readability and review.
Another example is all of the theme data migrations. The theme data constructor requires supplying a ton of parameters, which would be a ton of code to write just to validate one parameter changed, and easy to miss or introduce mistakes since other transformations like the first example cannot be subject to analysis.

Also, I think .expect is the required format in order for the tests to work.

I see. I'm trying to minimize the amount of non-runnable code produced. I'll look at creating a more realistic test file (e.g. with multiple deprecated/renamed properties) which only includes fully automatic fixes.
Not being to analyze a file because it doesn't have a Dart extension is a separate issue, I'll look for that in dart-lang. Thanks


Add Favorites collection

Favorites collection has email of user as the id, with an array of all the user favorites items


added possibility to skip cache lookup per project or globally

added possibility to skip generated sources restore on per-project level
added some logging
Following this checklist to help us incorporate your
contribution quickly and easily:

[ ] Make sure there is a JIRA issue filed
for the change (usually before you start working on it).  Trivial changes like typos do not
require a JIRA issue.  Your pull request should address just this issue, without
pulling in other changes.
[ ] Each commit in the pull request should have a meaningful subject line and body.
[ ] Format the pull request title like [MNG-XXX] - Fixes bug in ApproximateQuantiles,
where you replace MNG-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
[ ] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Run mvn clean verify to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.
[ ] You have run the Core IT successfully.

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.
To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

[ ] I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

[ ] In any other case, please file an Apache Individual Contributor License Agreement.

Sorry, I do not have an MNG number yet to add as a proper commit comment
But basically this PR is about adding 2 new features which are required to use this extension in yet another complex multi-module project in Deutsche Bank

ability to skip cache lookup for a particular module or whole project (required to get some modules always built e.g. via a profile - e.g. I have some additional artifacts published by CI). Alternatively it allows an easy "force" rebuild of a whole project
The difference with not using cache is that results are put into cache, just current matching versions are not taken from caches

ability to disable generated source restoration on a per-module level. This is required to overcome issues with some weird Maven plugins which go nuts when cached generated sources are present and produce incorrect build result

@eltsovalex https://issues.apache.org/jira/projects/MBUILDCACHE for this project

is it possible to add a note in documentation about this feature and how to use it, please?
https://github.com/apache/maven-build-cache-extension/tree/master/src/site/markdown

Typos fixed in javadoc + new parameters described in documentation


[Flow] Attestation support

For logical flows, physical flows and physical specifications using the new attestation table

Add Flow - Explicit attestation
Add data type decorator - implicit attestation of the logical and physical flows and spec
delete an entity - remove the attestation also


Manually set refreshing

When calling setRefreshing(true) by layout.post()ing a Runnable, it set's the refresh direction of the last refresh. Is it possible to manually set the direction of this post?

You can set it using setDirection I think


Added improved loader with maputnik logo

Uses loader animation from https://github.com/maputnik/design/blob/master/logos/logo-loading.svg
Note: the palette dots animate, see demo below

Demo https://2232-84182601-gh.circle-artifacts.com/0/artifacts/build/index.html#0.36/0/0
Note: this is only for first load, I see the loader quite a bit during dev and wanted to do this for a while  😁

Nice @orangemug - out of curiosity: Did you create the SVG by hand or did you use an editor?

Thanks @pathmapper so its a bit of both. How the maputnik/designs repo works is you have a 'base' SVG that was created in inkscape src/logos/_includes/base.svg off of the raster that @lukasmartinelli originally created src/reference/logo.png. Then the other files reference the parts in that file via
<!-- you can change the style in the SVG <use/> -->
<use
  xlink:href="_includes/base.svg#map__main"
  fill="#4eba6f"
  stroke="#000000"
/>

If you start a webserver in the root of the maputnik/designs repo (you can use python -m SimpleHTTPServer 8000) all the logos in https://github.com/maputnik/design/tree/master/src/logos will render correctly when opened in the browser. Because external file references when using <use xlink:href="..." /> don't work everywhere because of browser security (an example is in an <img /> tag) the build script ./scripts/build resolves/inlines the xlink:href in the <use /> tag.
Note: Yes I should really document that webserver step in the maputnik/designs README.md, on my list.
I think it works ok, and although a little confusing to start with I think it's a really nice setup reusing assets out of a 'base' file. All the animation is done with CSS from within the derived SVG files (for example https://github.com/maputnik/design/tree/master/src/logos). I don't think I would setup anything that was likely to change that much this way as it's probably overboard, but for something like a logo where the base design never changes I'm pretty happy with how it works. Although I'd love if an open source SVG editor supported this sort of setup directly.
It's also worth noting that setting this all up took a while, but I really just wanted to understand SVG a little more so it was as much a learning experience as anything else 😁

Merging!

Thanks a lot @orangemug for the explanation!


Fixed issue 3359

Fixed issue #3359 so that empty grade entry item won't trigger validation error when creating or editing grade entry form.

Hi @czack810150, are you sure this is exactly the check you want? My understanding is params[:grade_entry_items] is actually a list of grade entry items (since there are multiple "column subforms" on the main form, one per column); if this is the case, you should really filter out the elements that are blank in the list, rather than dropping the list itself. You could do this in the update_grade_entry_form_params helper.


Make it easier to provide admin user passwords

 Issue by sym3tri
Friday Jun 12, 2015 at 18:56 GMT
Originally opened as https://github.com/coreos-inc/auth/issues/256

Creating users with the overlord API accepts password hashes only, but there is no utility provided to hash the password. We should make things easier on the user and allow them to post plaintext password and/or provide a utility to create a hash from a plaintext password.

Bootstrapping API is unfortunately useless without this feature.

passwords can now be provided statically.


Import STIX file or stream [FeatureRequest]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Multiple IOC's provided by various third party organisations require manual work to pull out required OIC's (IP's, file hashes etc)
Describe the solution you'd like
A clear and concise description of what you want to happen.
It would be great if we could utilise the STIX format in LME to introduce IOC's provided by say CISA for example.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Only manually pull out the data which is not really feasible when there are hundreds of items.
Additional context
Add any other context or screenshots about the feature request here.
For example the STIX file provided by CISA relating to the current threat situation due to the Russian/Ukrainian situation would be very useful. For example https://www.cisa.gov/uscert/sites/default/files/publications/AR21-013A.stix.xml

Closed due to project archive


Bad interaction between super and old-style classes

super fails to work correctly (for __init__ at least) when old style classes are parents. In this code:
class aco(object):
  def __init__(self):
    print("init aco");
    super(aco, self).__init__();
    print("post super aco")

class x0:
  def foo():
    return

class x1(object):
  def bar():
    return

class two(x0, aco, x1):
  def __init__(self):
    print("init two");
    super(two, self).__init__();
    print("post super two")

when you created an object of class two you would expect it to call the __init__ method from its parent aco. In fact, the super call seems to go nowhere and only the "init two" and "post super two" strings are printed. Note that when x0 is turned into a new class the code works as expected. It's also okay when the order is changed to have aco precede x0.
We've traced the problem to some suspicious handling of old classes in Super.cs in the method below. Note the commented out line with +++ and the following line which contains the suggested fix. We are unfamiliar enough with the code nor do we have an extensive enough test suite to submit this as a pull request but can do so if requested.
        private bool TryLookupInBase(CodeContext context, PythonType pt, string name, object self, out object value) {
            PythonTypeSlot dts;

            if (pt.OldClass == null) {
                // new-style class, or reflected type, lookup slot
                if (pt.TryLookupSlot(context, name, out dts) &&
                    dts.TryGetValue(context, self, DescriptorContext, out value)) {
                    return true;
                }
            } else {
                // old-style class, lookup attribute
                OldClass dt = pt.OldClass;

                if (
                    // +++ PythonOps.TryGetBoundAttr(context, dt, name, out value)
                    dt.TryLookupOneSlot(DescriptorContext, name, out value) // suggested fix
                    ) {
                    value = OldClass.GetOldStyleDescriptor(context, value, self, DescriptorContext);
                    return true;
                }
            }
            value = null;
            return false;
        }

Note: this problem turned up using the Facebook Marketing API via their supported Python SDK.

Thanks for the report. Created a PR based on the suggested fix and added a test.


docs(firebase_messaging): import statement for cloud_messaging

Description
Import from firebase_messaging instead of cloud_messaging
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[ ] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[ ] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

Looks like this was already fixed in another commit. Thank you for the PR though 🎉


Support for TRVZB

Certainly not a priority, but perhaps TRVZB support can be added any soon? Now, apart from visibility, nothing can be done with TRVZB.
Best regards.

#1244


Lawnicons Guidelines v2.2

Adjusted the requirements for square icons, gave samples and clarified wording with @Grabstertv.
Lawnicons Guidelines in Figma.
Type of change
:x: Bug fix (non-breaking change which fixes an issue)
:white_check_mark: General change (non-breaking change that doesn't fit the above categories, such as copyediting)

All that's left is to find time for a review.

Thanks! @Chefski @Grabstertv @KTibow


Cannot copy text shown in the Log

Especially important for error reporting.

Relavant to SUPPORT-2753.


Adding Open Images v7 to the zoo

What changes are proposed in this pull request?
Added open images v7 support to our model zoo. It can be loaded from the model zoo with
dataset = foz.load_zoo_dataset('open-images-v7'). The only difference from open images v6 is
support for "point" labels, which are incorporated via the Keypoints Label class. Support for open images v6 is unchanged, despite the backend details being shuffled around.
When the dataset is loaded with the "points" label, each point is represented by a separate Keypoint. I chose this approach because the points in the open images dataset have non-uniform numbers of yes/no votes and methods of generation.
The one detail I'm not sure about is that currently, dataset.info is carrying along a point_classes key. This is because there were points with class labels that were not in the classes_map, but were in the point_classes metadata. Unsure if we should pass this through to the user, or just hide it internally.
What areas of FiftyOne does this PR affect?

[ ] App: FiftyOne application changes
[ ] Build: Build and test infrastructure changes
[x] Core: Core fiftyone Python library changes
[x] Documentation: FiftyOne documentation changes
[ ] Other

@brimoor just updated the Open Images tutorial to get the bare bones point labels incorporated and explained in there. The tutorial threw an error because I don't have tensorflow in this particular conda env, but I didn't change any of the code that goes into that part of the tutorial.
Let me know if you think we should include the type of point IoU evaluation they introduced in the paper https://arxiv.org/pdf/2210.14142v2.pdf or if we think this is sufficient

@brimoor just updated. Made all of the suggested edits/improvements except for the _parse_label_types() direction. Docs build as desired

Hmm, renaming the OpenImagesV6Dataset type will break existing open-images-v6 datasets people have since we store this in the ~/fiftyone/open-images-v6/info.json:
{
    "name": "open-images-v6",
    "zoo_dataset": "fiftyone.zoo.datasets.base.OpenImagesV6Dataset",
    "dataset_type": "fiftyone.types.dataset_types.OpenImagesV6Dataset",
    "num_samples": 1676,
    "downloaded_splits": {
        "validation": {
            "split": "validation",
            "num_samples": 250
        },
        "test": {
            "split": "test",
            "num_samples": 253
        },
        "train": {
            "split": "train",
            "num_samples": 1173
        }
    },
    "classes": [
        "Accordion",
        "Adhesive tape",
        "Aircraft",
...

So it will try to load that dataset type and not find it.

Hmm, renaming the OpenImagesV6Dataset type will break existing open-images-v6 datasets people have since we store this in the ~/fiftyone/open-images-v6/info.json:
So it will try to load that dataset type and not find it.

@jacobmarks FYI there actually a place in the code where we explicitly account for historical name changes like this. You can just add a check at the place linked below that will replace OpenImagesV6Dataset with OpenImagesDataset in the dataset type:
https://github.com/voxel51/fiftyone/blob/f6fea3686c032a942f182ff835651dd19ab26b2e/fiftyone/zoo/datasets/__init__.py#L1294

@brimoor which approach would you prefer? I have it working with OpenImagesV6Dataset and OpenImagesV7Dataset, but I'm happy to do the historical name change you suggested. Either way

@brimoor which approach would you prefer? I have it working with OpenImagesV6Dataset and OpenImagesV7Dataset, but I'm happy to do the historical name change you suggested. Either way

Either is fine. Mostly stylistic. A name like OpenImagesDataset implies that we intend to support loading all future versions of the dataset using that type. That is how COCODetectionDataset works, for example. Loads both COCO-2014 and COCO-2017 datasets.

@brimoor which approach would you prefer? I have it working with OpenImagesV6Dataset and OpenImagesV7Dataset, but I'm happy to do the historical name change you suggested. Either way

Either is fine. Mostly stylistic. A name like OpenImagesDataset implies that we intend to support loading all future versions of the dataset using that type. That is how COCODetectionDataset works, for example. Loads both COCO-2014 and COCO-2017 datasets.

@brimoor in general I think I like your suggestion of having a future-facing OpenImagesDataset, but given the issue with version parsing and how the solution we had involved just looking for "points.csv", maybe distinguishing OpenImagesDatasetV6 and OpenImagesDatasetV7 is the right call?


How to prioritize topics in Kafkajs

In Java I can prioritise one topic over another creating 2 consumers each listening to 1 of my 2 topics. Then loop over my consumer in priority order and call consumer1.poll, for example.
Is there a way to do something similar with kafkajs since there is no poll() or fetch() method?
Thanks,
Thomas

I would be curious to know what you mean by prioritizing in this case. Some pseudo-code showing what you mean would be helpful. From your description I don't really understand how one topic would be prioritized over the other in any meaningful sense:
for (const consumer of consumers) {
  const records = await consumer.poll()
  for (const record of records) {
    await processRecord(record)
  }
}

In the example above, one consumer isn't prioritized over the other - it just happens to be polled before the other. They will still be using the same amount of resources and have the same throughput and latency. To prioritize one of the other you'd need to do something more complex like measure the amount of time/resources spent and only fetch with the second producer if there's enough time/resources available once the first has finished. Something like:
const startTime = process.hrtime.bigint()
const records = await prioritizedConsumer.poll()
await processRecords(records)
const duration = process.hrtime(start) - startTime

if (TIME_BUDGET - duration >= THRESHOLD) {
  const records = await lessPrioritizedConsumer.poll()
  await processRecords(records)
}

Something like that could be achieved by using pause & resume

Hi Nevon,
In java you could achieve that by creating a list of consumers ordered by priority and do something like that:

I am not sure if something like that is possible with Kafkajs, because I think there is no function such as consumer.poll() returning the list of records to check if the list empty or not. But I am quite new with Kafkajs, so please tell me if I am wrong and how I can do that.
If not possible, then I will try what you propose using pause & resume function.
Let me know if this is not clear.
Thanks!

The code you just took a screenshot of is roughly equivalent to my first example above, but as I said, there's nothing about this that makes it prioritized. It's still just going to poll with one consumer, process those messages, then poll the next consumer and process those messages, etc. That's not prioritization. Both consumers are getting the exact same resources and time. Whether you poll consumer A and then B or B and then A doesn't matter, they will both have the exact same throughput and latency over time.
Let's simplify it with an example:
const counters = new Map();

const createWorker = (name) => () => {
  for (let i = 0; i < 10; i++) {
    const current = counters.get(name) ?? 0
    counters.set(name, current + 1)
  }
}

const workers = [createWorker('A'), createWorker('B')]

for (let i = 0; i <10; i++) {
  for (const worker of workers) {
    worker()
  }
}

Once this runs, both A and B are going to have processed 100 records. If I swap the order of them, they're still going to have processed 100 records. If you toss this into an infinite loop, whenever you stop the loop they are going to have processed approximately the same number of records - one is not prioritized over the other.

Well, this is just some pseudo-code. You could probably do it in a nicer way. It's true that on startup you wouldn't have all the partitions in there, so you'd start out consuming from the less prioritized topic as well.
One way to make it more robust is to listen to the GROUP_JOIN event, which contains memberAssignment, which are all the topic-partitions that are assigned to the consumer. so you could keep track of which partitions are assigned, and then in the END_BATCH_PROCESS event see if all the assigned partitions have had 0-size batches.

I'll close this as it doesn't refer to an issue with KafkaJS itself. Feel free to re-open if needed.


Add throughput timer configuration

The new "timers" section describes configuration for different timers.
Specifically, in the "throughput" section, it is possible to disable the throughput timer (enabled by default). This allows to avoid the performance degradation whenever the throughput measurement is not needed, for example in production environment.
No device synchronize() is invoked when "synchronized" is set to False (default is True). This allows to produce approximate throughput measurements with minimal performance penalty.

We've discovered the following issues in the current implementation of the Throughput timer:

The timer invokes synchronize() twice on each step at start and  stop.

Calling synchronize() ensures that all workloads running on the device complete.
This makes sure that the measured time includes both the elapsed CPU and the device times.

However, being an expensive operation, synchronize() causes the performance drop of up to 20-25%
on some accelerators (e.g. HPU). On the other hand, that avoiding the call to synchronize()
still produces a meaningful performance estimate without performance drop.

Since there is no configuration flag to disable the timer, today a user pays consistent performance penalty,
even in cases when the throughput measurement may be unnecessary (e.g. production).

The purpose of this PR is to add config options to control the above behavior.

@loadams @mrwyattii @lekurile can this change be merged?

@loadams @mrwyattii @lekurile can this change be merged?

Sorry for the delay @nelyahu - I will make sure this is merged shortly.


MI - DEVELOPER section (Developer Kick Start/Tutorial)

In EI650, we had a tutorial like[1] page, which walked through the entire developer workflow.
In EI7, keeping the same content as 'Developer Kick Start'[2] to be consistent with other topics in the section and elsewhere in the docs.
Please see preview on netlify[3].
[1] https://docs.wso2.com/display/EI650/Developing+Cloud+Native+Integration
[2] https://github.com/wso2/docs-ei/blob/master/en/micro-integrator/docs/develop/integration-development-kickstart.md
[3] https://wso2docs-configurationcatalog-3.netlify.com/develop/integration-development-kickstart/

Closing this issue as the content was verified for EI650. Will do the follow up changes after testing with EI7.


Consider disallowing tagged templates with tags returning type predicate types

Basically the following is kind of absurd:
function tag(strs: any, x: any): x is number {
    return typeof x === "number";
}

let x = Math.random() < 0.5 ? 100 : "100"; // x: string | number
if (tag `Hello ${ x } world`) {
    // x: number
    // But why didn't you just use a function call?
    x.toExponential();
}

This is the dual suggestion of #3554: see https://github.com/Microsoft/TypeScript/issues/3554#issuecomment-113865503.

I do agree that this is absurd, and hence do not see the value of fixing it. inclined to resolve this as won't fix.

@mhegazy this issue is for disallowing it instead of silently failing.

It is not failing. the guard will not work. similarly to typeof x == "string" does not work, and there is no warning for it either. if we hear enough complains about it we can restrict it. i do not see much probability of the two features being used in conjunction in real world scenarios.


Deduplicate LDAP auto-attributes

Codecov Report

Merging #186 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff          @@
##           master   #186   +/-   ##
=====================================
  Coverage     100%   100%
=====================================
  Files          11     11
  Lines        1306   1306
=====================================
  Hits         1306   1306

Impacted Files
Coverage Δ

ldap2pg/validators.py
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01a413d...12629e9. Read the comment docs.


make error during libmolgrid step

Issue summary
I'm trying to compile gnina v1.1 on a fresh Ubuntu 22.04 wsl installation, but the make step fails with an error that I can't interpret.
[ 84%] Built target test_gridinterp_cpp
[ 86%] Building CXX object test/CMakeFiles/test_mgrid_cpp.dir/test_mgrid.cpp.o
In file included from /usr/local/cuda-12/include/cub/util_device.cuh:52,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/util.h:48,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/internal/copy_cross_system.h:49,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/copy.h:111,
                 from /usr/local/cuda-12/include/thrust/system/detail/adl/copy.h:50,
                 from /usr/local/cuda-12/include/thrust/detail/copy.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/copy.h:98,
                 from /usr/local/cuda-12/include/thrust/detail/allocator/copy_construct_range.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/allocator/copy_construct_range.h:53,
                 from /usr/local/cuda-12/include/thrust/detail/contiguous_storage.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/contiguous_storage.h:243,
                 from /usr/local/cuda-12/include/thrust/detail/temporary_array.h:47,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/scan_by_key.inl:35,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/scan_by_key.h:150,
                 from /usr/local/cuda-12/include/thrust/detail/scan.inl:32,
                 from /usr/local/cuda-12/include/thrust/scan.h:1664,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/reduce_by_key.inl:40,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/reduce_by_key.h:95,
                 from /usr/local/cuda-12/include/thrust/detail/reduce.inl:33,
                 from /usr/local/cuda-12/include/thrust/reduce.h:789,
                 from /home/christian/Download/gnina/build/libmolgrid-prefix/src/libmolgrid/test/test_mgrid.cpp:10:
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘void cub::CUB_200400___CUDA_ARCH_LIST___NS::CTA_SYNC()’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:271:5: error: ‘__syncthreads’ was not declared in this scope
  271 |     __syncthreads();
      |     ^~~~~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::CTA_SYNC_AND(int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:280:12: error: ‘__syncthreads_and’ was not declared in this scope
  280 |     return __syncthreads_and(p);
      |            ^~~~~~~~~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::CTA_SYNC_OR(int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:289:12: error: ‘__syncthreads_or’ was not declared in this scope
  289 |     return __syncthreads_or(p);
      |            ^~~~~~~~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘void cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_SYNC(unsigned int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:298:5: error: ‘__syncwarp’ was not declared in this scope
  298 |     __syncwarp(member_mask);
      |     ^~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_ANY(int, unsigned int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:307:12: error: ‘__any_sync’ was not declared in this scope
  307 |     return __any_sync(member_mask, predicate);
      |            ^~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_ALL(int, unsigned int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:316:12: error: ‘__all_sync’ was not declared in this scope
  316 |     return __all_sync(member_mask, predicate);
      |            ^~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_BALLOT(int, unsigned int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:325:12: error: ‘__ballot_sync’ was not declared in this scope
  325 |     return __ballot_sync(member_mask, predicate);
      |            ^~~~~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘unsigned int cub::CUB_200400___CUDA_ARCH_LIST___NS::SHFL_IDX_SYNC(unsigned int, int, unsigned int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:368:12: error: ‘__shfl_sync’ was not declared in this scope
  368 |     return __shfl_sync(member_mask, word, src_lane);
      |            ^~~~~~~~~~~
/usr/local/cuda-12/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::RowMajorTid(int, int, int)’:
/usr/local/cuda-12/include/cub/util_ptx.cuh:415:39: error: ‘threadIdx’ was not declared in this scope
  415 |     return ((block_dim_z == 1) ? 0 : (threadIdx.z * block_dim_x * block_dim_y)) +
      |                                       ^~~~~~~~~
In file included from /usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/barrier_cluster.h:30,
                 from /usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx.h:74,
                 from /usr/local/cuda-12/include/cuda/ptx:19,
                 from /usr/local/cuda-12/include/cuda/discard_memory:25,
                 from /usr/local/cuda-12/include/cub/util_device.cuh:57,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/util.h:48,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/internal/copy_cross_system.h:49,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/copy.h:111,
                 from /usr/local/cuda-12/include/thrust/system/detail/adl/copy.h:50,
                 from /usr/local/cuda-12/include/thrust/detail/copy.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/copy.h:98,
                 from /usr/local/cuda-12/include/thrust/detail/allocator/copy_construct_range.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/allocator/copy_construct_range.h:53,
                 from /usr/local/cuda-12/include/thrust/detail/contiguous_storage.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/contiguous_storage.h:243,
                 from /usr/local/cuda-12/include/thrust/detail/temporary_array.h:47,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/scan_by_key.inl:35,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/scan_by_key.h:150,
                 from /usr/local/cuda-12/include/thrust/detail/scan.inl:32,
                 from /usr/local/cuda-12/include/thrust/scan.h:1664,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/reduce_by_key.inl:40,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/reduce_by_key.h:95,
                 from /usr/local/cuda-12/include/thrust/detail/reduce.inl:33,
                 from /usr/local/cuda-12/include/thrust/reduce.h:789,
                 from /home/christian/Download/gnina/build/libmolgrid-prefix/src/libmolgrid/test/test_mgrid.cpp:10:
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘uint32_t cuda::ptx::__4::__as_ptr_smem(const void*)’:
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:40:44: error: ‘__cvta_generic_to_shared’ was not declared in this scope
   40 |   return static_cast<_CUDA_VSTD::uint32_t>(__cvta_generic_to_shared(__ptr));
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘uint64_t cuda::ptx::__4::__as_ptr_gmem(const void*)’:
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:60:44: error: ‘__cvta_generic_to_global’ was not declared in this scope
   60 |   return static_cast<_CUDA_VSTD::uint64_t>(__cvta_generic_to_global(__ptr));
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘_Tp* cuda::ptx::__4::__from_ptr_smem(size_t)’:
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:73:33: error: there are no arguments to ‘__cvta_shared_to_generic’ that depend on a template parameter, so a declaration of ‘__cvta_shared_to_generic’ must be available [-fpermissive]
   73 |   return reinterpret_cast<_Tp*>(__cvta_shared_to_generic(__ptr));
      |                                 ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:73:33: note: (if you use ‘-fpermissive’, G++ will accept your code, but allowing the use of an undeclared name is deprecated)
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘_Tp* cuda::ptx::__4::__from_ptr_gmem(size_t)’:
/usr/local/cuda-12/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:94:33: error: there are no arguments to ‘__cvta_global_to_generic’ that depend on a template parameter, so a declaration of ‘__cvta_global_to_generic’ must be available [-fpermissive]
   94 |   return reinterpret_cast<_Tp*>(__cvta_global_to_generic(__ptr));
      |                                 ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/local/cuda-12/include/thrust/system/cuda/detail/util.h:48,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/internal/copy_cross_system.h:49,
                 from /usr/local/cuda-12/include/thrust/system/cuda/detail/copy.h:111,
                 from /usr/local/cuda-12/include/thrust/system/detail/adl/copy.h:50,
                 from /usr/local/cuda-12/include/thrust/detail/copy.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/copy.h:98,
                 from /usr/local/cuda-12/include/thrust/detail/allocator/copy_construct_range.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/allocator/copy_construct_range.h:53,
                 from /usr/local/cuda-12/include/thrust/detail/contiguous_storage.inl:31,
                 from /usr/local/cuda-12/include/thrust/detail/contiguous_storage.h:243,
                 from /usr/local/cuda-12/include/thrust/detail/temporary_array.h:47,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/scan_by_key.inl:35,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/scan_by_key.h:150,
                 from /usr/local/cuda-12/include/thrust/detail/scan.inl:32,
                 from /usr/local/cuda-12/include/thrust/scan.h:1664,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/reduce_by_key.inl:40,
                 from /usr/local/cuda-12/include/thrust/system/detail/generic/reduce_by_key.h:95,
                 from /usr/local/cuda-12/include/thrust/detail/reduce.inl:33,
                 from /usr/local/cuda-12/include/thrust/reduce.h:789,
                 from /home/christian/Download/gnina/build/libmolgrid-prefix/src/libmolgrid/test/test_mgrid.cpp:10:
/usr/local/cuda-12/include/cub/util_device.cuh: In static member function ‘static typename AgentT::TempStorage& cub::CUB_200400___CUDA_ARCH_LIST___NS::detail::vsmem_helper_impl<AgentT>::get_temp_storage(cub::CUB_200400___CUDA_ARCH_LIST___NS::NullType&, cub::CUB_200400___CUDA_ARCH_LIST___NS::detail::vsmem_t&)’:
/usr/local/cuda-12/include/cub/util_device.cuh:160:63: error: ‘blockIdx’ was not declared in this scope
  160 |       static_cast<char*>(vsmem.gmem_ptr) + (vsmem_per_block * blockIdx.x));
      |                                                               ^~~~~~~~
/usr/local/cuda-12/include/cub/util_device.cuh: In static member function ‘static bool cub::CUB_200400___CUDA_ARCH_LIST___NS::detail::vsmem_helper_impl<AgentT>::discard_temp_storage(typename AgentT::TempStorage&)’:
/usr/local/cuda-12/include/cub/util_device.cuh:201:38: error: ‘threadIdx’ was not declared in this scope
  201 |     const std::size_t linear_tid   = threadIdx.x;
      |                                      ^~~~~~~~~
/usr/local/cuda-12/include/cub/util_device.cuh:202:50: error: ‘blockDim’ was not declared in this scope
  202 |     const std::size_t block_stride = line_size * blockDim.x;
      |                                                  ^~~~~~~~
make[5]: *** [test/CMakeFiles/test_mgrid_cpp.dir/build.make:76: test/CMakeFiles/test_mgrid_cpp.dir/test_mgrid.cpp.o] Error 1
make[4]: *** [CMakeFiles/Makefile2:1164: test/CMakeFiles/test_mgrid_cpp.dir/all] Error 2
make[3]: *** [Makefile:146: all] Error 2
make[2]: *** [CMakeFiles/libmolgrid.dir/build.make:86: libmolgrid-prefix/src/libmolgrid-stamp/libmolgrid-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:414: CMakeFiles/libmolgrid.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

Steps to reproduce
git clone https://github.com/gnina/gnina.git
cd gnina
git checkout v1.1
mkdir build
cd build
cmake ..
make

Your system configuration
Operating system: wsl2 install of Ubuntu 22.04.3 LTS
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- The CUDA compiler identification is NVIDIA 12.5.82
-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0
--   Git               :   v1.1
--   System            :   Linux
--   C++ compiler      :   /usr/bin/c++
--   Release CXX flags :   -O3 -DNDEBUG -Wno-deprecated-declarations -Wno-unknown-pragmas -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Debug CXX flags   :   -g -Wno-deprecated-declarations -Wno-unknown-pragmas -fPIC -Wall -Wno-sign-compare -Wno-uninitialized
--   Build type        :   Release
--
--   BUILD_SHARED_LIBS :   ON
--   BUILD_python      :   ON
--   BUILD_matlab      :   OFF
--   BUILD_docs        :   ON
--   CPU_ONLY          :   OFF
--   USE_OPENCV        :   0
--   USE_LEVELDB       :   0
--   USE_LMDB          :   0
--   USE_NCCL          :   OFF
--   ALLOW_LMDB_NOLOCK :   OFF
--
-- Dependencies:
--   BLAS              :   Yes (Atlas)
--   Boost             :   Yes (ver. 1.74)
--   glog              :   Yes
--   gflags            :   Yes
--   OpenBabel         :   No
--   protobuf          :   Yes (ver. ..)
--   CUDA              :   Yes (ver. 12.5)
--
-- NVIDIA CUDA:
--   Target GPU(s)     :   Auto
--   GPU arch(s)       :   sm_61
--   cuDNN             :   Yes (ver. 9.2.1)
--
-- Python:
--   Interpreter       :   /usr/bin/python3 (ver. 3.10.12)
--   Libraries         :   /usr/lib/x86_64-linux-gnu/libpython3.10.so (ver 3.10.12)
--   NumPy             :   /usr/lib/python3/dist-packages/numpy/core/include (ver 1.21.5)
--
-- Documentaion:
--   Doxygen           :   /usr/bin/doxygen (1.9.1)
--   config_file       :   /home/christian/Download/gnina/caffe/.Doxyfile
--
-- Install:
--   Install path      :   /usr/local

Thanks for the bug report.  It looks like CUDA 12.5 changed how thrust includes were processed, breaking things.  I have updated the molgrid code to be compatible with 12.5.  A fresh checkout should work now.

Thanks! It does proceed now, but fails a bit later on. Is this the same problem?
Consolidate compiler generated dependencies of target caffe
[  0%] Building CXX object caffe/src/caffe/CMakeFiles/caffe.dir/layers/cudnn_softmax_layer.cpp.o
In file included from /usr/local/cuda-12.5/include/cub/util_device.cuh:52,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/util.h:48,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/internal/copy_cross_system.h:49,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/copy.h:111,
                 from /usr/local/cuda-12.5/include/thrust/system/detail/adl/copy.h:50,
                 from /usr/local/cuda-12.5/include/thrust/detail/copy.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/copy.h:98,
                 from /usr/local/cuda-12.5/include/thrust/detail/allocator/copy_construct_range.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/allocator/copy_construct_range.h:53,
                 from /usr/local/cuda-12.5/include/thrust/detail/contiguous_storage.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/contiguous_storage.h:243,
                 from /usr/local/cuda-12.5/include/thrust/detail/vector_base.h:39,
                 from /usr/local/cuda-12.5/include/thrust/device_vector.h:34,
                 from /home/christian/Download/gnina/caffe/src/caffe/layers/cudnn_softmax_layer.cpp:4:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘void cub::CUB_200400___CUDA_ARCH_LIST___NS::CTA_SYNC()’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:271:5: error: ‘__syncthreads’ was not declared in this scope
  271 |     __syncthreads();
      |     ^~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::CTA_SYNC_AND(int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:280:12: error: ‘__syncthreads_and’ was not declared in this scope
  280 |     return __syncthreads_and(p);
      |            ^~~~~~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::CTA_SYNC_OR(int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:289:12: error: ‘__syncthreads_or’ was not declared in this scope
  289 |     return __syncthreads_or(p);
      |            ^~~~~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘void cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_SYNC(unsigned int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:298:5: error: ‘__syncwarp’ was not declared in this scope
  298 |     __syncwarp(member_mask);
      |     ^~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_ANY(int, unsigned int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:307:12: error: ‘__any_sync’ was not declared in this scope
  307 |     return __any_sync(member_mask, predicate);
      |            ^~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_ALL(int, unsigned int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:316:12: error: ‘__all_sync’ was not declared in this scope
  316 |     return __all_sync(member_mask, predicate);
      |            ^~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::WARP_BALLOT(int, unsigned int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:325:12: error: ‘__ballot_sync’ was not declared in this scope
  325 |     return __ballot_sync(member_mask, predicate);
      |            ^~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘unsigned int cub::CUB_200400___CUDA_ARCH_LIST___NS::SHFL_IDX_SYNC(unsigned int, int, unsigned int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:368:12: error: ‘__shfl_sync’ was not declared in this scope
  368 |     return __shfl_sync(member_mask, word, src_lane);
      |            ^~~~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_ptx.cuh: In function ‘int cub::CUB_200400___CUDA_ARCH_LIST___NS::RowMajorTid(int, int, int)’:
/usr/local/cuda-12.5/include/cub/util_ptx.cuh:415:39: error: ‘threadIdx’ was not declared in this scope
  415 |     return ((block_dim_z == 1) ? 0 : (threadIdx.z * block_dim_x * block_dim_y)) +
      |                                       ^~~~~~~~~
In file included from /usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/barrier_cluster.h:30,
                 from /usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx.h:74,
                 from /usr/local/cuda-12.5/include/cuda/ptx:19,
                 from /usr/local/cuda-12.5/include/cuda/discard_memory:25,
                 from /usr/local/cuda-12.5/include/cub/util_device.cuh:57,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/util.h:48,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/internal/copy_cross_system.h:49,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/copy.h:111,
                 from /usr/local/cuda-12.5/include/thrust/system/detail/adl/copy.h:50,
                 from /usr/local/cuda-12.5/include/thrust/detail/copy.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/copy.h:98,
                 from /usr/local/cuda-12.5/include/thrust/detail/allocator/copy_construct_range.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/allocator/copy_construct_range.h:53,
                 from /usr/local/cuda-12.5/include/thrust/detail/contiguous_storage.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/contiguous_storage.h:243,
                 from /usr/local/cuda-12.5/include/thrust/detail/vector_base.h:39,
                 from /usr/local/cuda-12.5/include/thrust/device_vector.h:34,
                 from /home/christian/Download/gnina/caffe/src/caffe/layers/cudnn_softmax_layer.cpp:4:
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘uint32_t cuda::ptx::__4::__as_ptr_smem(const void*)’:
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:40:44: error: ‘__cvta_generic_to_shared’ was not declared in this scope
   40 |   return static_cast<_CUDA_VSTD::uint32_t>(__cvta_generic_to_shared(__ptr));
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘uint64_t cuda::ptx::__4::__as_ptr_gmem(const void*)’:
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:60:44: error: ‘__cvta_generic_to_global’ was not declared in this scope
   60 |   return static_cast<_CUDA_VSTD::uint64_t>(__cvta_generic_to_global(__ptr));
      |                                            ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘_Tp* cuda::ptx::__4::__from_ptr_smem(size_t)’:
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:73:33: error: there are no arguments to ‘__cvta_shared_to_generic’ that depend on a template parameter, so a declaration of ‘__cvta_shared_to_generic’ must be available [-fpermissive]
   73 |   return reinterpret_cast<_Tp*>(__cvta_shared_to_generic(__ptr));
      |                                 ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:73:33: note: (if you use ‘-fpermissive’, G++ will accept your code, but allowing the use of an undeclared name is deprecated)
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h: In function ‘_Tp* cuda::ptx::__4::__from_ptr_gmem(size_t)’:
/usr/local/cuda-12.5/include/cuda/std/detail/libcxx/include/__cuda/ptx/instructions/../ptx_helper_functions.h:94:33: error: there are no arguments to ‘__cvta_global_to_generic’ that depend on a template parameter, so a declaration of ‘__cvta_global_to_generic’ must be available [-fpermissive]
   94 |   return reinterpret_cast<_Tp*>(__cvta_global_to_generic(__ptr));
      |                                 ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/util.h:48,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/internal/copy_cross_system.h:49,
                 from /usr/local/cuda-12.5/include/thrust/system/cuda/detail/copy.h:111,
                 from /usr/local/cuda-12.5/include/thrust/system/detail/adl/copy.h:50,
                 from /usr/local/cuda-12.5/include/thrust/detail/copy.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/copy.h:98,
                 from /usr/local/cuda-12.5/include/thrust/detail/allocator/copy_construct_range.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/allocator/copy_construct_range.h:53,
                 from /usr/local/cuda-12.5/include/thrust/detail/contiguous_storage.inl:31,
                 from /usr/local/cuda-12.5/include/thrust/detail/contiguous_storage.h:243,
                 from /usr/local/cuda-12.5/include/thrust/detail/vector_base.h:39,
                 from /usr/local/cuda-12.5/include/thrust/device_vector.h:34,
                 from /home/christian/Download/gnina/caffe/src/caffe/layers/cudnn_softmax_layer.cpp:4:
/usr/local/cuda-12.5/include/cub/util_device.cuh: In static member function ‘static typename AgentT::TempStorage& cub::CUB_200400___CUDA_ARCH_LIST___NS::detail::vsmem_helper_impl<AgentT>::get_temp_storage(cub::CUB_200400___CUDA_ARCH_LIST___NS::NullType&, cub::CUB_200400___CUDA_ARCH_LIST___NS::detail::vsmem_t&)’:
/usr/local/cuda-12.5/include/cub/util_device.cuh:160:63: error: ‘blockIdx’ was not declared in this scope
  160 |       static_cast<char*>(vsmem.gmem_ptr) + (vsmem_per_block * blockIdx.x));
      |                                                               ^~~~~~~~
/usr/local/cuda-12.5/include/cub/util_device.cuh: In static member function ‘static bool cub::CUB_200400___CUDA_ARCH_LIST___NS::detail::vsmem_helper_impl<AgentT>::discard_temp_storage(typename AgentT::TempStorage&)’:
/usr/local/cuda-12.5/include/cub/util_device.cuh:201:38: error: ‘threadIdx’ was not declared in this scope
  201 |     const std::size_t linear_tid   = threadIdx.x;
      |                                      ^~~~~~~~~
/usr/local/cuda-12.5/include/cub/util_device.cuh:202:50: error: ‘blockDim’ was not declared in this scope
  202 |     const std::size_t block_stride = line_size * blockDim.x;
      |                                                  ^~~~~~~~
make[2]: *** [caffe/src/caffe/CMakeFiles/caffe.dir/build.make:468: caffe/src/caffe/CMakeFiles/caffe.dir/layers/cudnn_softmax_layer.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1325: caffe/src/caffe/CMakeFiles/caffe.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

Yes, this is in issue within the caffe library, which is no longer part of gnina.  If you remove the line #include "thrust/device_vector.h" from gnina/caffe/src/caffe/layers/cudnn_softmax_layer.cpp it will compile with CUDA 12.5.

Awesome. I ran into one more bump - for some reason /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.74.0 was missing from the following files
caffe/tools/CMakeFiles/extract_features.dir/link.txt
caffe/tools/CMakeFiles/upgrade_net_proto_binary.dir/link.txt
caffe/tools/CMakeFiles/upgrade_net_proto_text.dir/link.txt
caffe/tools/CMakeFiles/upgrade_solver_proto_text.dir/link.txt
caffe/examples/CMakeFiles/convert_cifar_data.dir/link.txt

But adding the link allowed me to finish the installation.
Thanks so much for your help, David.


Gisgraphy Geolocalisation service

Hi Anton
I would like to know if it currently possible to use Gisgraphy Geo-localisation service to get nearest location or landmark to trackers on tracker, i just made a new setup of Gisgraphy and i'm looking to use it as most places out of the cities in my country are not properly mapped.

Please be more specific. Provide some coordinates, what Gisgraphy returns, and what's expected.


Add Rap God by Eminem

Song Name
Rap God
Artist Name
Eminem
Source
Youtube
Youtube Link
XbGs_qK2PQA
Direct File Link
https://www.github.com/rapgod.mp3
Song ID
955695
Start Offset [ms]
0
Extra
Created from Auto Nong.
Created for level: 111844538
Downloaded: No

{
"name": "Rap God",
"artist": "Eminem",
"source": "youtube",
"startOffset": 0,
"yt-id": "XbGs_qK2PQA",
"songs": [
955695
]
}


Bug in Promise.prototype._resolveCallback?

I'm looking at the source code for bluebird v3.3.5. Think I might have noticed a typo/bug.
In Promise.prototype._resolveCallback, there is the following code:
var bitField = promise._bitField;
if (BIT_FIELD_CHECK(IS_PENDING_AND_WAITING_NEG)) {
    ...
} else if (BIT_FIELD_CHECK(IS_FULFILLED)) {
    ...
} else if (BIT_FIELD_CHECK(IS_REJECTED)) {
    ...
}

https://github.com/petkaantonov/bluebird/blob/master/src/promise.js#L464
The bitfield variable is never referred to after it's defined, and the following lines call BIT_FIELD_CHECK() with no bitField argument.
I'm guessing the code above was intended to read:
var bitField = promise._bitField;
if (BIT_FIELD_CHECK(IS_PENDING_AND_WAITING_NEG, bitField)) {
    ...
} else if (BIT_FIELD_CHECK(IS_FULFILLED, bitField)) {
    ...
} else if (BIT_FIELD_CHECK(IS_REJECTED, bitField)) {
    ...
}

... but I have no idea of the implications of this change!

Those are not functions but macros which the build step expands

@petkaantonov Sorry my mistake. Bluebird is a tour-de-force but the code can be pretty hard to follow for the uninitiated with all the optimizations it contains.


Update reference-connect-version-history.md

Fixing minor typos.

@NuAlex : Thanks for your contribution! The author(s) have been notified to review your proposed change.

#label:"aq-pr-triaged"

#sign-off

#sign-off

@billmath
This PR has developed merge conflicts in the following file:

articles/active-directory/hybrid/reference-connect-version-history.md

Pull requests can't be reviewed or merged when there are merge conflicts. After you resolve the conflicts, you can sign off again.
For help with resolving merge conflicts, refer to the Contributor Guide, or try the Support channel in Teams.
Thanks!
#hold-off

This has been fixed.
#please-close


Comiple error Afnetworking 3.0 Swift

When i have written below code in my project ia m getting compile error.
Cannot convert value of type (NSurlsessionTask!,AnyObject!)->to expected argument type ((NSurlsessionDataTask,ANyObject?)->Void)
manager.GET(url, parameters: myDic1, success: {
            (operation: NSURLSessionTask!, responseObject: AnyObject!) in
            let responseDict = responseObject as! Dictionary<String, AnyObject>
            if responseDict["status"] as! String == "success"
            {

                successClosure(responseObject as? NSDictionary, APIResult.APISuccess)

            }
            else
            {
                successClosure(responseObject as? NSDictionary, APIResult.APIError)

            }

            }, failure: {
                (operation: NSURLSessionTask!, error: NSError!) in
                let errorDict = ["error": error]
                successClosure(errorDict as NSDictionary, APIResult.APIFail)

        })

It looks like you are using NSURLSessionTask instead of NSURLSessionDataTask.

Same error persist with NSURLSessionDataTask

Looks like you've got your optionals wrong. This is a more basic Swift question, I think.

Facing the same issue how to resolve it ?

Did anyone figure this one out?


[Suggestion/support] for wardenloot

would it be possible to add support for warden loot mod OR the ability to convert the armor back to Normal netherite
because warden loot in order to make the warden set requires a set of netherite armor but doesnt register gilded netherite armor.

Maybe you shouldn't be able to upgrade gilded armor any further. Do you think it's better to add gilded depth-keeper's armor?


Create Windows Executable File

The windows executable is not feasible as pyinstaller gives error  on dependent package casparser .
A better solution which is cloud based has been implemented


Small English typos

Coverage remained the same at 100.0% when pulling 2925989bbaf3ece01d79d90d9bf8dfe8f766dbd7 on braco:patch-1 into 3538d1493507d4418382c5c41be704bb19a58f43 on epoberezkin:master.


[ML/AI] Digital Twins and IoT for Robotics

Proposed title of article
Digital Twins and IoT for Robotics
Proposed article introduction
A digital twin is a virtual representation of a physical product or process, used to understand and predict the physical counterpart’s performance characteristics. Digital twins provide valuable data on actual motion vs. commanded motion. This information is ideally suited for predictive analytics, Artificial Intelligence (AI) training, decision making, Augmented Reality (AR) interaction, and teleoperation.
Robots and digital twins can also be used to gather data or monitor environments in dangerous situations without the risk to human or animal life. Robots can be driven via VR controls, and its sensors can be used to detect things beyond the sensory capacities of any live-form.
Key takeaways
Why digital twin makes sense to predictive analytics.
How IoT and digital twin are enhancing teleoperation.
How robotics are benefiting from digital twins and IoT implementation.
Using IoT and digital twins to reduce costs.
Article quality
This article discusses digital twins in relation to IoT and robotics, something that other blogs have not.
References
Please list links to any published content/research that you intend to use to support/guide this article.

Thank you for submitting your topic suggestion. There seems to be many articles that already cover this topic. Can you tell me how yours will build upon them and be unique to add value to our readers?

@louisefindlay23
This article explains the use of digital twins in predictive analytics and teleoperation, which is not available in other articles. In addition, it explores how a combination of digital twins and IoT technologies can enhance robotic operations and help in cost reductions.

Closed #6014 via #6117


FIX: file extension bug

The current version doesn't correctly detect the file extension. Using split('.')[0] only works for filenames without any dot (.) in the name and where the first dot is the separater between filename and file extension.
This PR takes care of this issue by checking for specific MRI file extensions.

Codecov Report

Merging #84 into master will decrease coverage by 0.94%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##           master      #84      +/-   ##
==========================================
- Coverage   92.04%   91.09%   -0.95%
==========================================
  Files           6        6
  Lines         377      382       +5
==========================================
+ Hits          347      348       +1
- Misses         30       34       +4

Impacted Files
Coverage Δ

atlasreader/atlasreader.py
96% <33.33%> (-1.56%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc52b3c...9b3a8ce. Read the comment docs.


Add CompanyExportCountry model

This replaces the previous PR https://github.com/uktrade/data-hub-leeloo/pull/2085

Add data migration to migrate future_interest_countries to
CompanyExportCountry.
Change API future_interest_countries field to use this new model
Ignore these objects during company merge (replicates current
behaviour)
Add set_user_edited_export_countries method to perform logic when user
sets these countries
Add get_active_future_export_countries method to query the new model
and return a list of active countries of all sources.

Description of change
Add a new model which can record future interest countries from a variety of sources.
This model is populated from the old future_interest_country m2m using a INSERT INTO... SELECT.
The two main sources of information are 'user' and 'external'. User represents those countries manually entered by the user on the front-end. External represents those countries that only came in from an import from the data science pipeline.
The key difference is that we want the user to be able to delete countries that came in from data science, and have those deletions stick when the import is refreshed. On the other hand, data science imports should only be able to add new countries, not delete user-entered countries or override any that users have deleted.
Checklist

[x] Has a new newsfragment been created? Check changelog/README.md for instructions
[ ] Do any added or updated endpoints appear in the API documentation? See docs/Maintaining the API documentation.md for more details
[x] Have any relevant search models been updated?
[x] Have any relevant fixtures (fixtures/test_data.yaml) been updated?
[x] Have any relevant select-/prefetch-related field lists in the views and search apps been updated?
[x] Has the admin site been updated (for new models, fields etc.)?
[ ] Has the README been updated (if needed)?

The general approach looks OK to me, but the PR is a bit large at just over 900 lines added (the guideline is up to 400).
Can I suggest splitting it to make reviewing easier? For example, first add the new model and the underlying logic, and then do the data migration and plug it into the API etc. (Feature flags can also be used if needed.)

@reupen only if it is 100% necessary. I understand the line limit as an aspirational goal, but this has been dragging on for so long already.

Split into https://github.com/uktrade/data-hub-leeloo/pull/2177/files and https://github.com/uktrade/data-hub-leeloo/pull/2178


LLM return non exist action in predefined list

What problem or use case are you trying to solve?
Maybe user proposed problem like
'str' object has no attribute 'copy'

in #718, #1064, #1126. I should better figure out a way to improve it, or find a workaround to pass it.
Because it depends on the LLM response, I am still trying to find a way to reproduce it. Want to get the error LLM response to do futher investigation.
Describe the UX of the solution you'd like
Do you have thoughts on the technical implementation?

Solution 1: From backend logic, can we add some retry logic if LLM return action not in predefine list?
Solution 2:When initialization the agent, can we add some prompts emphasize the action type? I have seen some case emphasize it in prompts can help. But currently we have no way to do experiments to evaluate it.
Solution 3: In solution 1, when we find error and retry, maybe we can add extra prompts to let LLM pay attention to providing a correct action in list.

Describe alternatives you've considered
Additional context

I still cannot reproduce it locally. I have no chance to see the true response. Our code didn't log response, so no user can provide it. If some one can provide the true error response message, that will be very helpful.

When initialization the agent, can we add some prompts emphasize the action type?

I found https://www.youtube.com/watch?v=RJ6NN8Y-xok&ab_channel=OfirPress very worth watching. At 18:58, the author mentioned that they had to emphasize the current directory the agent is on, otherwise it would often mess up.
That being said, apparently powerful LLMs have no problem of remembering what actions are legal. Maybe you want to tune prompt only for "very weak" LLMs, but we don't have a framework to tune prompt per LLM (group) yet.

When initialization the agent, can we add some prompts emphasize the action type?

I found https://www.youtube.com/watch?v=RJ6NN8Y-xok&ab_channel=OfirPress very worth watching. At 18:58, the author mentioned that they had to emphasize the current directory the agent is on, otherwise it would often mess up.
That being said, apparently powerful LLMs have no problem of remembering what actions are legal. Maybe you want to tune prompt only for "very weak" LLMs, but we don't have a framework to tune prompt per LLM (group) yet.

Yeah, I also see this case. This is what trigger me idea. I wonder whether add some redundant info in prompts can be helpful to solve it.


Option to config python lib path from env var

The simplest way to give the option to configure python library path from environment variables, targeting the issue https://github.com/tensorflow/serving/issues/216 about a scriptable complete configuration for tensorflow + serving.

Can one of the admins verify this patch?

@googlebot Done.

Jenkins, test this please.


i18n(es): update middleware

Description (required)
updates middleware

Related issues & labels (optional)

Closes # N/A
Suggested label: i18n

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

es
guides/middleware.mdx
Localization changed, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.

Thank you for the review @liruifengv 🫡

I'm closing this PR since it got stale and outdated, feel free to open it again later!


Add proxies_config to botocore config

This is a continuation of #2219. Adding a subset of possible urllib3 configs to botocore's configuration mechanisms. This will allow AWS CLI to take advantage of these changes as well.

Codecov Report

Merging #2240 (cf97096) into develop (96f9740) will decrease coverage by 0.42%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##           develop    #2240      +/-   ##
===========================================
- Coverage    98.25%   97.83%   -0.43%
===========================================
  Files           58       58
  Lines        10857    10868      +11
===========================================
- Hits         10668    10633      -35
- Misses         189      235      +46

Impacted Files
Coverage Δ

botocore/args.py
100.00% <ø> (ø)

botocore/config.py
100.00% <ø> (ø)

botocore/endpoint.py
98.03% <ø> (ø)

botocore/httpsession.py
92.93% <100.00%> (-0.13%)
:arrow_down:

botocore/compat.py
72.15% <0.00%> (-22.16%)
:arrow_down:

botocore/hooks.py
95.18% <0.00%> (-0.81%)
:arrow_down:

botocore/awsrequest.py
97.22% <0.00%> (-0.35%)
:arrow_down:

botocore/utils.py
97.55% <0.00%> (-0.19%)
:arrow_down:

botocore/credentials.py
98.54% <0.00%> (-0.11%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96f9740...15c5d54. Read the comment docs.

Documentation questions, since I've been asked to update the "Shared Configuration and Credentials" guide:

The proxies setting is not covered in this documentation, despite having been added something like four years ago. Since proxies_config works with that, does that mean that both need to be documented in this guide or neither do?
If this does indeed need to be added to the shared configuration guide, I could use some guidance in terms of which services the proxy configuration apply to.

I can't really do much until I'm clear on these points.


onSelectSlot in Firefox

Hello!
How can I get global object "window" or "window.event" in Firefox, if onSelectSlot event hasn't "event" argument
I call "window.event.target.className" inside onSelectSlot function and browser throw me an error, what window.event is undefined

window.event is non-standard and not supported by Firefox


Query compilation fails with an error about Timestamp for a query/data set with no timestamps

This occurs on the old Mongo connector.
With this test data:
{ "b": { "ba": true, "bb": ["x", false, { "foo": "bar" }], "bc": 0.5 } }
{ "b": [1, 2, 3] }
{ "b": "test flat value" }

And this query:
SELECT
    CASE TYPE_OF(t.b)
    WHEN "map" THEN t.b{*}
    WHEN "array" THEN LENGTH(t.b)
    ELSE t.b END as b
FROM
    `/test/flatten_test.json` as t

The resulting error is:
{
  "error": {
    "status": "Unable to compile to JavaScript.",
    "detail": {
      "message": "Operation/value could not be compiled to JavaScript: Timestamp(1901-12-13T20:45:52Z, 0)",
      "value": "Timestamp(1901-12-13T20:45:52Z, 0)",
      "logicalplan": {
        "type": "LogicalPlan/Let",
        "label": "'__tmp0",
        "children": [
          {
            "type": "LogicalPlan/Let",
            "label": "'__tmp1",
            "children": [
              {
                "type": "LogicalPlan/Read",
                "label": "/test/flatten_test.json"
              },
              {
                "type": "LogicalPlan/Typecheck",
                "label": "Obj(Map[], Some(Top))",
                "children": [
                  {
                    "type": "LogicalPlan/Free",
                    "label": "'__tmp1"
                  },
                  {
                    "type": "LogicalPlan/Free",
                    "label": "'__tmp1"
                  },
                  {
                    "type": "LogicalPlan/Constant",
                    "label": "NA"
                  }
                ]
              }
            ]
          },
          {
            "type": "LogicalPlan/Invoke",
            "label": "Squash",
            "children": [
              {
                "type": "LogicalPlan/Invoke",
                "label": "MakeObject",
                "children": [
                  {
                    "type": "LogicalPlan/Constant/Str",
                    "label": "\"b\""
                  },
                  {
                    "type": "LogicalPlan/Let",
                    "label": "'__tmp2",
                    "children": [
                      {
                        "type": "LogicalPlan/ObjectProject",
                        "label": "Free('__tmp0){\"b\"}"
                      },
                      {
                        "type": "LogicalPlan/Typecheck",
                        "label": "Str",
                        "children": [
                          {
                            "type": "LogicalPlan/Free",
                            "label": "'__tmp2"
                          },
                          {
                            "type": "LogicalPlan/Invoke",
                            "label": "Cond",
                            "children": [
                              {
                                "type": "LogicalPlan/Invoke",
                                "label": "Eq",
                                "children": [
                                  {
                                    "type": "LogicalPlan/Invoke",
                                    "label": "unknown function: Returns the simple type of a value.",
                                    "children": [
                                      {
                                        "type": "LogicalPlan/ObjectProject",
                                        "label": "Free('__tmp0){\"b\"}"
                                      }
                                    ]
                                  },
                                  {
                                    "type": "LogicalPlan/Constant/Str",
                                    "label": "\"map\""
                                  }
                                ]
                              },
                              {
                                "type": "LogicalPlan/Invoke",
                                "label": "FlattenMap",
                                "children": [
                                  {
                                    "type": "LogicalPlan/Let",
                                    "label": "'__tmp3",
                                    "children": [
                                      {
                                        "type": "LogicalPlan/ObjectProject",
                                        "label": "Free('__tmp0){\"b\"}"
                                      },
                                      {
                                        "type": "LogicalPlan/Typecheck",
                                        "label": "Obj(Map[], Some(Top))",
                                        "children": [
                                          {
                                            "type": "LogicalPlan/Free",
                                            "label": "'__tmp3"
                                          },
                                          {
                                            "type": "LogicalPlan/Free",
                                            "label": "'__tmp3"
                                          },
                                          {
                                            "type": "LogicalPlan/Constant",
                                            "label": "Obj(Map( -> NA))"
                                          }
                                        ]
                                      }
                                    ]
                                  }
                                ]
                              },
                              {
                                "type": "LogicalPlan/Invoke",
                                "label": "Cond",
                                "children": [
                                  {
                                    "type": "LogicalPlan/Invoke",
                                    "label": "Eq",
                                    "children": [
                                      {
                                        "type": "LogicalPlan/Invoke",
                                        "label": "unknown function: Returns the simple type of a value.",
                                        "children": [
                                          {
                                            "type": "LogicalPlan/ObjectProject",
                                            "label": "Free('__tmp0){\"b\"}"
                                          }
                                        ]
                                      },
                                      {
                                        "type": "LogicalPlan/Constant/Str",
                                        "label": "\"array\""
                                      }
                                    ]
                                  },
                                  {
                                    "type": "LogicalPlan/Invoke",
                                    "label": "Length",
                                    "children": [
                                      {
                                        "type": "LogicalPlan/Free",
                                        "label": "'__tmp2"
                                      }
                                    ]
                                  },
                                  {
                                    "type": "LogicalPlan/ObjectProject",
                                    "label": "Free('__tmp0){\"b\"}"
                                  }
                                ]
                              }
                            ]
                          },
                          {
                            "type": "LogicalPlan/Constant",
                            "label": "NA"
                          }
                        ]
                      }
                    ]
                  }
                ]
              }
            ]
          }
        ]
      }
    }
  }
}

I think the Timestamp is showing up because the way we do typechecks in Mongo Expressions (since it has no way to actually check types) is abusing its total ordering of values. So we generate a bunch of values at the boundaries of different types and see how the value you’re checking compares to them.
The problem that shows up here is that we sometimes need to convert Mongo Expressions into JavaScript, and that is only a partial function, and the semantics don’t line up great (e.g., we have to identify when we’re using comparisons to fake type checking and convert it to actual type checks in JS).
Thankfully, I just yesterday committed a PR against the new Mongo branch that eliminates all Expression => JS conversions. Instead, we now have Ior[Js, ExprOp] as a result from compiling MapFunc (it used to be Js \/ ExprOp), so if we ever need to “convert” to JS we just grab the Js element from the Ior rather than trying to recover it from the Expression.

We just need to verify once newMaster is available.

The query compiles without error against next-major.

Fixed via #2730