Configuration re-resolution, take 2

This replaces most of #403. It's a stripped down version, without IDE tests or tests for replacing dependencies.
There are still some tests failing:

[x] ProjectDependencyResolveIntegrationTest. resolved project artifacts contain project version in their names (ignored for now)
[x] ConfigurationOnDemandIntegrationTest.respects buildProjectDependencies setting
[x] Antlr2PluginIntegrationTest.uses antlr v2 if no explicit dependency is set
[x] AntlrPluginIntegrationTest.can apply plugin unqualified
[x] AntlrPluginIntegrationTest.plugin can build with empty project
[x] AntlrPluginIntegrationTest.plugin does not force creation of build dir during configuration
[x] MavenConversionIntegrationTest.multiModuleWithRemoteParent
[x] TestTaskIntegrationTest.should not resolve configurations when there are no tests

Let's have a take 3 in PR #425.


Getting a build error

No resource identifier found for attribute 'appComponentFactory'

+1

Could you please provide a stacktrace or more detailed error message?


Task: List v2: Implement hashed key

In list v2 every row has to be identified by a unique key, the key can by any unique value on the list data.

If no key is selected then the whole list data item has to be hashed using non-cryprographic algorithm (for fastest hash generation) to generate unique keys for each row.
A warning also should be added to the property depicting the importance of the property.

Issues added [here](https://github.com/appsmithorg/appsmith/pull/18017#issuecomment-1301832070 .)


Missing parts of the API for repository issues

Motivation: see issue #286 and https://github.com/agda/agda/pull/5691
Closes: #286 and #287
Needs: release (e.g. 0.28)
As observed in #286 and #287, the API for listing repository issues, see https://docs.github.com/en/rest/reference/issues#list-repository-issues, is incompletely reflected in github-0.27.  Esp. the functionality to get just the issues for a particular milestone is not exported.
This PR adds the missing parts of the API as new methods to construct IssueRepoMod.
With the functionality added by this PR, the API is now complete.  In particular, I conjecture the following completeness statement:

Using the language exported by GitHub.Data.Options,
for all o : IssueRepoOptions, there is some m : IssueRepoMod, such that o = toIssueRepoOptions m.

In contrast, the API of 0.27 is incomplete:

There is are no m : IssueRepoMod and i : Id Milestone such that toIssueRepoOptions m = defaultIssueRepoOptions{ issueRepoOptionsMilestone = FilterBy i }.

Successful CI run at: https://github.com/andreasabel/github/actions/runs/1556147789

Ping @phadej.  The workflow needs to be started manually.

Thanks for the review, @phadej!  I updated the PR accordingly.

@phadej, what's the plan?  Can you release this fix?

@andreasabel do you want to be come the maintainer of github. I'll happily transfer this repository to you.

@andreasabel do you want to be come the maintainer of github. I'll be happy to transfer this repository to you.
(I have been looking for someone to look after this library for quite some time)

@phadej Yes, I am happy to maintain github.  I have some interest in this library.

@phadej ping.

I cannot transfer this repository as andreasabel/github already exists

Looks like that's not enough, it still errors with

andreasabel already has a repository in the phadej/github network


pragma once in main file except in header file

ALEInfo:
Current Filetype: cpp
Available Linters: ['cppcheck', 'g++', 'cppcheck', 'g++']
Enabled Linters: ['cppcheck', 'g++', 'cppcheck', 'g++']
This occurs when #pragma once is in a header file (.h) in C++. I'm guessing that the issue is that the clang linter assumes that the file being checked is a main file (.cpp) when ALE is running on the current file. Is there a way to filter the linter's response based on file name and file type?

Moved to #392.

I finally fixed this one. I just needed to check the file extension for the open file and filter that one error out.


fix: hashtag filter for Unicode characters

This commit addresses an issue #2016  with the hashtag filter, which was not working correctly for hashtags containing Unicode characters. The filter was only capturing single-letter hashtags instead of complete hashtags.

@boojack I haven't touched the Go file, but the backend test is failing. Is there any way to debug the issue?


Fix InputHandler for MS Edge.

The latest IE -> EDGE change broke InputHandler...

I signed it!
(As Remember The Milk Inc.)

I'm not seeing "omarkilani" as having signed the CLA. Can you sign it with this user name?

@joeltine: I honestly don't know what that means -- we have a corporate CLA that incorporates a Google Apps group with a bunch of email addresses in it. My commit email is in that group, but it wasn't tied to my GitHub account -- I've added that email to GitHub too... Does that fix it?
@concavelenz Yup.
As an aside, I feel like people are going to spend a lot of time re-adding all the IE checks back for Edge -- I obviously don't know how much testing is being done internally over there but it may have been better  (?) to just s/IE/EDGE_OR_IE/ when EDGE was introduced and then removed the hacks as it was proven they were no longer required.
As far as I can see, Edge still has all of the baggage associated with IE (execCommand, etc) and most of the same bugs. The layout engine even seems to need the exact same CSS hacks as IE 11.

You were the not the first to state that IE Edge seems closer to IE 11 than Webkit/Blink and we had a long debate about this internally to Google.  There were two camps those that were for following the MS recommendation of treating MS Edge as "webkit" and those that wanted to threat it as "ie".  What we now  is Edge as separate layout engine.  I would like to know what they actually changed to be closer to webkit.

@concavelenz Well... they could have fooled me. I guess it's early days but unless there's some X-UA-Compatible setting that I'm missing... I don't see how it's anything like Blink/WebKit.
Not to mention the glacial JS engine on real world apps, but that's another story entirely. I may be slightly jaded... :smile:


losing multiple cursors when saving

VSCode Version: Code 1.17.2 (b813d12980308015bcd2b3a2f6efa5c810c33ba5, 2017-10-16T13:47:41.889Z)
OS Version: Darwin x64 16.7.0
Extensions:

Extension
Author (truncated)
Version

compulim-vscode-closetag
Com
1.1.0

sublime-commands
Zar
0.1.0

prettier-vscode
esb
0.24.0

sublime-keybindings
ms-
2.9.1

expand-selection-to-scope
vit
0.2.0

sublime-text-pack
wad
0.0.2

Steps to Reproduce:

in a doc with some text that repeats
select one of the repeating instances
Cmd+D multiple times to select all repeating instances
multiple cursors/selections are created
change to something else (type 'abc')

Expected result:
Changes are saved and cursors are maintained.
Actual result:
Only first cursor remains.

Reproduces without extensions: could not launch code without extensions.

@mgiraldo I cannot reproduce in a .txt file. Are you perhaps seeing this only in certain file types ?

that's true. i'm using React. i cannot reproduce in txt either. try saving this code in js format:
import React from "react";
import fetch from "isomorphic-fetch";

import MainLayout from "../../components/MainLayout";
import AllExhibitions from "../../components/ExhibitionsComponents/AllExhibitions";
import Footer from "../../components/ExhibitionsComponents/Footer";
import { getCurrentUrl } from "utilFunctions";
import {
  EXHIBITS_ENDPOINT,
  EXHIBIT_PAGES_ENDPOINT,
  FILES_ENDPOINT
} from "constants/exhibitions";

const Exhibitions = ({ url, exhibitions }) =>
  <MainLayout route={url}>
    <AllExhibitions route={url} exhibitions={exhibitions} />
    <Footer />
  </MainLayout>;

Exhibitions.getInitialProps = async ({ req }) => {
  const currentUrl = getCurrentUrl(req);
  const exhibitsRes = await fetch(`${currentUrl}${EXHIBITS_ENDPOINT}`);
  const exhibitsJson = await exhibitsRes.json();
  const exhibitions = await Promise.all(
    exhibitsJson
      .map(async exhibit => {
        const exhibitPageRes = await fetch(
          `${currentUrl}${EXHIBIT_PAGES_ENDPOINT}?exhibit=${exhibit.id}`
        );
        const exhibitJson = await exhibitPageRes.json();

        const itemId = exhibitJson.find(
          exhibit =>
            exhibit.slug === "home-page" ||
            exhibit.slug === "homepage" ||
            exhibit.order === 0
        ).page_blocks[0].attachments[0].item.id;
        const filesRes = await fetch(
          `${currentUrl}${FILES_ENDPOINT}?item=${itemId}`
        );
        const filesJson = await filesRes.json();

        const thumbnailUrl = filesJson[0].file_urls.fullsize;
        return Object.assign({}, exhibit, { thumbnailUrl });
      })
      .reverse()
  );

  return { exhibitions };
};

export default Exhibitions;

@mgiraldo I have just tried with the code you've provided and I could not reproduce on a default installation of VS Code. I believe one of your installed extensions might be doing something on save (editing, formatting or something) to .js files ?

true... prettify does that… 🤔

You're right, I could reproduce this using editor.formatOnSave with the built-in TypeScript formatter.


Show completions for small types first

TypeScript Version:  nightly (2.5.0-dev.20170627)
Code
interface Point { x: number, y: number }
type Points = Point | Point[];
const x: Points = { /**/ }

Expected behavior:
Show completions for x and y first. Since that is the smallest type, it is the most likely to be implemented by an object literal.
Actual behavior:
They are all the way at the bottom. We get concat, copyWithin, and other properties of Array first.
Ref: https://github.com/Microsoft/TypeScript/issues/16781#issuecomment-311548696

I guess any primitive or literal type is strictly "smaller" than an object type, but is an object type "bigger" than a union of primitive types?

Also, we shouldn't show completions for literals, because no one in their right mind writes const x: string = { charAt(n) { ... }, charCodeAt(n) { ... }, ... }.

@andy-ms is this still an issue?

We did a workaround specifically for arrays in #18124. I guess we can wait and see if that's a good enough solution.

For this specific issue, i believe we did address the problem. closing for now. i do not think users interested in the original proposal will land here given the search keywords and the example.


Update timestamp.md

Add CURRENT_TIMESTAMP to the list

Thanks!

Looks like calling current_timestamp() does not work.
$ duckdb
v0.10.1 4a89d97db8
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D select current_timestamp();
Catalog Error: Scalar Function with name current_timestamp does not exist!
Did you mean "get_current_timestamp"?
LINE 1: select current_timestamp()
               ^
D select current_date() ;
┌────────────────┐
│ current_date() │
│      date      │
├────────────────┤
│ 2024-04-16     │
└────────────────┘
D

It's not a function, it's a magic value (which should probably be explicitly called out in the docs)

Thank you for quick roundturn.
-- works
select current_date=current_date();

-- fails and I think the function should be added just for consistency
select current_timestamp=current_timestamp();

It's not a function, it's a magic value (for lack of a better term, which should probably be explicitly called out in the docs)

@Mause: in my limited understanding, it should be foremost a Function() because it returns a system value, which then can be abbreviated by a "magic value" (which I would call a Time Key Expression).


Add CONTRIBUTING.md

Add CONTRIBUTING.md

Added Contributing.mg
https://github.com/NGimhana/24sevenoffice-php-adapter/tree/contributing_24sevenOffice-php-adapter


Add Ribir in areweguiyet

Ribir should be added to areweguiyet website to gain some attention!
areweguiyet contribution doc

Thanks, I will try it, when I have more time after release 0.4.


Ensure compatibility with latest IntelliJ

IntelliJ 2020 is out, so we need to check that the support plugin is still functioning correctly there. I've been using Metals recently, so help from any community members using IntelliJ would be greatly appreciated!

Im going to take a look right now.

So it looks like some binary compat issue was breaking nightlies for 2020.1.x but it should be fixed now.
If you saw this in your idea.log
2020-05-19 20:04:34,709 [ 108734]  ERROR - aemon.impl.PassExecutorService - Plugin to blame: Scala version: 2020.1.1174
2020-05-19 20:04:34,709 [ 108734]  ERROR - aemon.impl.PassExecutorService - 'org.jetbrains.plugins.scala.lang.psi.impl.toplevel.typedef.MixinN
odes$Map org.jetbrains.plugins.scala.lang.psi.impl.toplevel.typedef.TypeDefinitionMembers$.getTypes(com.intellij.psi.PsiClass)'
java.lang.NoSuchMethodError: 'org.jetbrains.plugins.scala.lang.psi.impl.toplevel.typedef.MixinNodes$Map org.jetbrains.plugins.scala.lang.psi.i
mpl.toplevel.typedef.TypeDefinitionMembers$.getTypes(com.intellij.psi.PsiClass)'
        at slinky.core.SlinkyInjector.createComponentBody(SlinkyInjector.scala:19)

this should be fixed soon per one of the developers on the intellij-scala team.

Thanks for investigating @dispalt! That's good to hear that things will be fixed in an upcoming release.

IntelliJ compatibility appears to be resolved, there are still a few issues with plugin detection, but that's an independent issue so closing.


Profile Service Issue

Hi, thanks for quick update.
There are some errors while sending request to card connect for GET and DELETE profile request. Please change the code on line 280 with this:

$res = $this->send('GET', "profile/$profile_id/$account_id/{$this->merchant_id}", null);

and on line 297 with

$res = $this->send('DELETE', "profile/$profile_id/$account_id/{$this->merchant_id}", null);

and also add this code in createProfile

$params = [
'merchid'  => $this->merchant_id
];
$request = array_merge($params, $request);

Hi @raghavdm,
I have added the changes you suggested to the profile and deleteProfile methods
In the future please submit a pull request as it's easier to see the changes you're requesting.
@lkagan addressed your request for defaulting the 'merchid' param in the createProfile method with his PR https://github.com/Dewbud/CardConnect/pull/3


Fixes duplicate and multiple locations with same id #171

Looks like sometimes the inverse is true and Location ID is present and Location isn't. Trying to replicate locally and then I'll push the change.

looks good. thanks!


Big tilesets with skewed ratio don't load correctly

Godot version
4.0.beta10.mono.official.d0398f62f
System information
Ubuntu 22.04.1 LTS
Issue description
It seems there is a limit in tileset size that causes a display problem or the root cause might be the unusual ratio (here 256x17024)

Steps to reproduce
Choose a big tileset and/or with huge Y size compared to X
Minimal reproduction project
I can't share the spritesheet (I bought it)

The texture is larger than 16384 on one of the axes, which is bound to cause various issues (especially on low-end hardware and mobile devices once you export the project). I suggest you rearrange the spritesheet to make it wider but less tall.

Thanks @Calinou, I was about to link a reddit comment where you mentioned the 16384 limit (and even 4096 for some devices!).
It would be useful to show a warning or even an error. It would avoid useless bug report to review ^^

Thanks @Calinou, I was about to link a reddit comment where you mentioned the 16384 limit (and even 4096 for some devices!).

Did this fix the issue for you?

Yes it worked but based on your reddit comment I'll use even smaller spritesheets, split by theme, to cap X and Y size at 4096.


system-probe: Allow specifying extra test arguments in invoke task

What does this PR do?
Allows extra go test arguments to be passed to the system-probe.test invoke task.

Motivation
To be able to pass things like -count=N to work around caching or to run tests multiple times. Instead of adding new wrapper arguments for each option of go test this allows arbitrary options to be passed.

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


add search functionality

It'd be nice to have a way to find a specific issue in a construction site (e.g. by ID) without having to navigate to its place in the hierarchy, which might get tedious.

merged into #24


HTML

HTML
User Story
As a bootcamp student
I want the prework notes to be structured on a web page
So I can easily find a read information
Acceptance Criteria
GIVEN a prework Study Guide website
WHEN I visit the website in my browser
THEN I see four boxes titled HTML, CSS, Git, and JavaScript with associated notes listed

There are some Listed Websites where you can check once :

freecodecamp
MDN WEB
codecademy
coursera
w3school
or last you can explore any repositories regading that on github.


How to read from DAI Medianizer / Uniswap price oracle

Due: 0000 24th April UTC +0

I can see this being at least 3 different blog posts:

How to read price of a token from Uniswap
How to read price of a token from MakerDAO Medianizer
How to read price of a token from Kyber

Does it need to be so specific?

I think it's a better idea for SEO.

If you don't want to, you can do a general one now, and I can do a more in-depth one later on.

SEO makes sense, lets hold off from DEX's from now and just do Maker's medianizer


Update GitHub workflow for release

We need to update the GitHub release_next workflow so that docs are published only after we successfully released a new version to npm. Right now, docs are published independently and could be deployed even if publishing a new version to npm failed, leading to discrepancy.

The desired flow is for docs to be published after publish_next

✅


Update updating-device-firmware-using-windows-update.md

Being a little more explicit about what it means by “device” to help avoid confusion

Thanks for the PR @Cymoki!


./whois: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by ./whois)

check your kernel version, https://go.dev/wiki/Linux


Add visually hidden info to learn more buttons

What would you like to share?
On the profile pages with milestones, there are several buttons/links which say learn more. This is not very helpful to screen reader users. We could add some visually hidden text so that screen reader users get more context about what they will learn more about when clicking on the button.
Additional information
No response

I think possible the solution could be:

Adding extra brief text with lean more
e.g. Learn more (visit Youtube channel/Github profile)

@emmalearnscode could you suggest other effective ways to achieve this.

There can be couple of things:

Add small tooltip popover for short info
Small one liner beside the button element.

Hi everyone 🙋🏻♂, I'm a newcomer trying to contribute in this OSS project and I was thinking if i can work on this issue. I'm planning on adding a small tooltip popover just as @msabirz suggested with the extra info as @87prashant mentioned.

Unfortunately a tooltip pop up will only make this more inaccessible.

Another way is we can have small 'i' icon which will have toggle functionality and on click it will show small desp.

Like @msabirz mentioned, that could be another way to do it. What I was doing was something like this, look:

Basically a small one liner describing where we're going (like on hover). But another solution would be something like this:

Or wouldn't it be better a "Check it out here" button instead of "Learn more"?

What do you think @emmalearnscode?

Ideally the buttons will say where the link leads as in the second image. But I’m not sure if we have something in the data that could be used already as the text.
‘Check it out here’ is no more useful than ‘learn more’. Imagine you cannot see the page and are cycling through a list of links. 5 links all titled ‘check it out here’ mean nothing without the visual context clues, just as ‘learn more’ needs visual context to be understood.

Ohh, I finally see what you mean. Reading some documentation of Prime React it seems that right now there is no accessibility support for the buttons. But I was thinking in using just a normal button and using the attribute aria-label for accessibility and work from there.
Regarding the text, I didn't find something in the data to use it as the displayed text. What I did was, basically doing something similar as the GetIcons.js component passing the icon name and with a switch function returned the required text, e.g. if the icon is github the displayed text would be "Learn more visiting the Github profile", etc.
Let me know what you think @emmalearnscode.

So far, what I've been doing is creating a normal button (no libraries) and making the UI of the button as similar as the prior button (primereact/button). Check it out:

How it looks like:

I'd like to know if I'm on the right track or if you have more suggestions @emmalearnscode

So far, what I've been doing is creating a normal button (no libraries) and making the UI of the button as similar as the prior button (primereact/button). Check it out:

How it looks like:

I'd like to know if I'm on the right track or if you have more suggestions @emmalearnscode

It’s an interesting idea but I think we will run into trouble with things like dollar, globe, book etc as that won’t make sense.
We are considering moving away from prime react although I think it should be possible to add an aria label. we should also be using links not buttons.
This is a tricky little problem, let’s keep thinking and discussing

You're right, it is possible to use aria-label attribute of the button just as it was before (my bad, didn't try that til' now). Now, I'm thinking of using a default text for things like dollar, globe, book, etc. The default text could be "Learn more on this website". It's just a thought, I'd like to know what you think.

hey everyone, apologies for my slow reply. The new version of the LinkFree project is out now using NextJS and TailWind
Now the whole block is clickable, not sure if this is better? I will close this issue and we can create a new one that is more relevant to the latest app version


Use binary search in getVisibleElements()

The method getVisibleElements() uses a linear search to find the first page and thumbnail in the visible range. It is called on every scroll event, so it can take a noticeable amount of time for the bottom pages of a very large document.
This PR replaces the linear search with a binary one (which needs O(log n) time).
A good test:
[3] pdf (5000+ pages)
A benchmark:
Scrolling through pages 5500 to 5515 of [3], using the ↓ key, Linux, Firefox Nightly 37.0a1, textLayer=off
Before:

After:

(Admittedly, the document has to be very large to notice a difference, but it should be more pronounced on slow devices).

@yurydelendik I addressed your comments.

/botio-linux preview

/botio unittest

/botio-windows unittest

Really good. Thank you for the patch.


Add permissions to trackerqueries

Allows only space collaborator (space owner is a collaborator as well) to create or delete trackerqueries

@sbose78 if all collaborators have contribute scope, then why do I need to check it explicitely? I already check if the user is a collaborator or not.

Thanks @baijum I will use the new Auth API and ask for review again.


Remove unused/unwanted link functionality from labels

When we created the Label component originally, we imagined that it could be something that was also a link. We never used it as such, and now we're confident that we won't want to. This removes the option.

Looks like some build errors though.

Looks like a snapshot needs updating. I'll get on it.


Updates and more links following "Is Vendor Prefixing Dead?"

Checked the table under https://css-tricks.com/is-vendor-prefixing-dead/#prefixing-in-2021 and updated a bit plus added links where applicable.
#5882, #5883 & #5887 should have been part of this too, sorry 0:-)
CSS3 font-kerning
Tested Safari on iOS only using 14.5.

Safari on iOS 14.5 screenshot

MDN says it's supported way longer though: https://developer.mozilla.org/en-US/docs/Web/CSS/font-kerning#browser_compatibility

Thanks! Just tested older iOS Safari and noticed proper unprefixed support starting in 12 so updated the PR with that. :)


GUVNOR-2698: row descriptions included in conversion

@jomarko Great job, thanks for the fix :-)

@manstis I will look on fails in the evening. Please be patient.

@manstis there were two problems:

problem with multiple tables in one xls file
tables with more than 2 data rows

Do you think can you have a look again? Sorry that my fix didn't come sooner.

@jomarko Looks good.. will wait for Jenkins to report a successful build before merging. Thanks!


patternlibrary_override_tags breaks all non-development installations

#8665 added a patternlibrary_override_tags tag library to wagtail.admin, which depends on the django-pattern-library package listed in our testing_extras dependencies. However, this tag library will be loaded on all environments that have wagtail.admin in their INSTALLED_APPS, so any Wagtail installation that doesn't include testing_extras is currently broken.
As a non-frontend dev, I don't know what this tag library is for, and there's no documentation for me to find out. Deleting it and uninstalling django-pattern-library doesn't break any tests, so it seems to me that it doesn't really belong in the testing dependencies. (If it really is a testing dependency, I'd suggest that moving patternlibrary_override_tags.py into one of the test apps in wagtail/test/ would let it do whatever it's meant to do without breaking non-testing environments.)

@gasman my apologies - I assumed that was installed as we use pattern library already.
To be honest - not super sure on how to work around this so maybe I should just revert that PR for now?
I might not have time to look into this fully sorry. But agree this should be documented correctly.
cc @thibaudcolas

PR up to revert for now https://github.com/wagtail/wagtail/pull/8800

Thanks for the info @gasman - will try to keep an eye out for that kind of issue again in the future.


Hyprland fails to start with malloc: corrupted top size

Hyprland Version
Hyprland, built from branch main at commit 68783d904d850df65887adb1bab7eff59943c1ac
Only happens in AUR package?
Bug or Regression?
Regression
Description
After updating from commit 3caaa483d4bf8f18f62f8a8f03a0922ea8a4cb7e to the latest (at time of posting), Hyprland fails to start with a malloc: corrupted top size.
Happens when starting it from sddm and tty.
Also found a segfault, see the backtrace in comments.
My config can be found at https://github.com/vypxl/dotfiles/tree/master/.config/hypr/
UPDATE
So the issue only seems to appear when using the executable produced by the PKGBUILD of the aur package.
How to reproduce
Just install hyprland-git from the aur and try to start it.
uname -a
Linux 6.6.3-1-MANJARO #1 SMP PREEMPT_DYNAMIC Tue Nov 28 20:37:45 UTC 2023 x86_64 GNU/Linux
Crash reports, logs, images, videos
See comments.

After updating from commit https://github.com/hyprwm/Hyprland/commit/3caaa483d4bf8f18f62f8a8f03a0922ea8a4cb7e to the latest

so it IS a regression. Why did you select bug?
Anyways, can't repro, so a git bisect would help.

So I am investigating.
It appears the issue does not occur when building the source manually, but when using the hyprland-git aur package.

mhm brilliant, then it's an issue with the package, no?

I will find out.

Though, a package bug should not cause Hyprland to crash without even a trace of why in the logs.

Managed to get a crash report by running in debug mode.
hyprlandCrashReport95760.txt

GDB Backtrace:
Thread 1 "Hyprland" received signal SIGSEGV, Segmentation fault.
0x00007ffff7e67c29 in wl_list_insert (list=0x0, elm=0x555556339ad0)
    at ../wayland-1.22.0/src/wayland-util.c:48
48		elm->next = list->next;
(gdb) bt
#0  0x00007ffff7e67c29 in wl_list_insert (list=0x0, elm=0x555556339ad0)
    at ../wayland-1.22.0/src/wayland-util.c:48
#1  0x0000555555695463 in wl_signal_add (listener=0x555556339ad0, signal=0x5555562ced18)
    at /usr/include/wayland-server-core.h:459
#2  addWLSignal(wl_signal*, wl_listener*, void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    (pSignal=0x5555562ced18, pListener=0x555556339ad0, pOwner=0x555556339680, ownerString="")
    at ../src/helpers/MiscFunctions.cpp:157
#3  0x000055555569c06f in CMonitor::onConnect(bool) (this=0x555556339680, noRule=false)
    at ../src/helpers/Monitor.cpp:35
#4  0x0000555555686676 in Events::listener_newOutput(wl_listener*, void*)
    (listener=<optimized out>, data=<optimized out>) at ../src/events/Monitors.cpp:88
#5  0x00007ffff7e6801e in wl_signal_emit_mutable (signal=<optimized out>, data=0x5555562cebf0)
    at ../wayland-1.22.0/src/wayland-server.c:2241
#6  0x00007ffff7e6801e in wl_signal_emit_mutable (signal=<optimized out>, data=0x5555562cebf0)
    at ../wayland-1.22.0/src/wayland-server.c:2241
#7  0x00007ffff7ec8d7a in wlr_wl_output_create () at /usr/lib/libwlroots.so.13032
#8  0x00007ffff7ec907d in  () at /usr/lib/libwlroots.so.13032
#9  0x00007ffff7ec9226 in  () at /usr/lib/libwlroots.so.13032
#10 0x00005555556006d1 in CCompositor::startCompositor() (this=<optimized out>)
    at ../src/Compositor.cpp:530
#11 0x00005555555e3f64 in main(int, char**) (argc=<optimized out>, argv=<optimized out>)
    at /usr/include/c++/13.2.1/bits/unique_ptr.h:199

odd, can you bisect it, if you can repro it?

Found the commit: d2c3b23a: deps: update wlroots
It seems the update to wlroots causes this. The aur package did not change for a few weeks. Building the same package from the commit right before d2c3b23a, namely b80c72c7, results in a working executable being installed.

So apparently Hyprland tries to call wl_signal_add
here. The signal struct is initialized, but the associated signal->listener_list.prev is a nullptr, causing a segmentation fault.

Hm
So somehow it magically fixed itself after I just rebuilt the package because I got frustrated.
Maybe it was some lingering incompatibility between wlroots versions.
Solution was: purge package and build cache and reinstall from scratch.


Containers on the same network overlay are inaccessible (1.12 rc4)

Output of docker version:
Docker version 1.12.0-rc4, build e4a0dbc

Output of docker info:
Containers: 10
 Running: 1
 Paused: 0
 Stopped: 9
Images: 18
Server Version: 1.12.0-rc4
Storage Driver: devicemapper
 Pool Name: docker-253:0-202338051-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 3.201 GB
 Data Space Total: 107.4 GB
 Data Space Available: 45.6 GB
 Metadata Space Used: 8.155 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.139 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null bridge host overlay
Swarm: active
 NodeID: 4ng2v259f4gc4k5ukxe1bt8lc
 IsManager: No
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 5.664 GiB
Name: vm-devdocke1.sseng.local
ID: 5D2F:EZVX:VNQB:ESML:NUM2:MLXZ:4L2L:347T:BRGX:AHWH:Y54X:NAEP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 <IP_ADDRESS>/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Two node swarm cluster, both centos vms created with HyperV. Hostname has been set to their internal FQDN [vmname].sseng.local. Hosts file has been updated to have self references and references to the the other VM. Iptables is set to allow all connections, firewalld is stopped.
Steps to reproduce the issue:

Set up a swarm with docker swarm init --listen-addr VM-DevDockMgr.sseng.local
Join the swarm with the worker node, also specifying --listen-addr VM-DevDockE1.sseng.local
Create the network overlay using docker network create -d overlay example
Create a zookeeper container using

docker service create --registry-auth --network example --name eventing-zookeeper $REGISTRY_PREFIX/eventing-zookeeper:$VERSION_EVENTING

5 Verify that the zookeeper container is running
Check the network
docker network inspect example
[
    {
        "Name": "example",
        "Id": "9ysv3eb2c5js6pjkp4h06na5l",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "<IP_ADDRESS>/24",
                    "Gateway": "<IP_ADDRESS>"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "6bf576f3a18c2a7cb0b6f7fe9ef03e525768fc644d88351e926c9fc3f4b4c554": {
                "Name": "eventing-zookeeper.1.4dtqkxvvu3nkoazp7x4v6tyt5",
                "EndpointID": "13a0c42d1bd9d50230652811febeab66de0b2ab1f8197426d4997d15ac6a48b4",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "<IP_ADDRESS>/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]

Inspect the process and verify that an IP was generated
 "NetworkSettings": {
     "Bridge": "",
     "SandboxID": "69969189ab93e2638be70cfe87172b0cc9817bc9e1eeacc62b6c077c6561d624",
     "HairpinMode": false,
     "LinkLocalIPv6Address": "",
     "LinkLocalIPv6PrefixLen": 0,
     "Ports": {
         "2181/tcp": null,
         "2888/tcp": null,
         "3888/tcp": null
     },
     "SandboxKey": "/var/run/docker/netns/69969189ab93",
     "SecondaryIPAddresses": null,
     "SecondaryIPv6Addresses": null,
     "EndpointID": "",
     "Gateway": "",
     "GlobalIPv6Address": "",
     "GlobalIPv6PrefixLen": 0,
     "IPAddress": "",
     "IPPrefixLen": 0,
     "IPv6Gateway": "",
     "MacAddress": "",
     "Networks": {
         "example": {
             "IPAMConfig": {
                 "IPv4Address": "<IP_ADDRESS>"
             },
             "Links": null,
             "Aliases": [
                 "6bf576f3a18c"
             ],
             "NetworkID": "9ysv3eb2c5js6pjkp4h06na5l",
             "EndpointID": "13a0c42d1bd9d50230652811febeab66de0b2ab1f8197426d4997d15ac6a48b4",
             "Gateway": "",
             "IPAddress": "<IP_ADDRESS>",
             "IPPrefixLen": 24,
             "IPv6Gateway": "",
             "GlobalIPv6Address": "",
             "GlobalIPv6PrefixLen": 0,
             "MacAddress": "02:42:0a:00:00:03"

6 Create the second service using
docker service create --registry-auth --network example --name eventing-kafka $REGISTRY_PREFIX/eventing-kafka:$VERSION_EVENTING

7 Check that the service is running and on the network from the second node
docker network inspect example
[
    {
        "Name": "example",
        "Id": "9ysv3eb2c5js6pjkp4h06na5l",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "<IP_ADDRESS>/24",
                    "Gateway": "<IP_ADDRESS>"
                }
            ]
        },
        "Internal": false,
        "Containers": {
            "d4069114e505e87cd0c2d805f03e5a70c103311e137ab40486c4d08198d75322": {
                "Name": "eventing-kafka.1.3eg70oqu2pc2ci8mfoiavzhbg",
                "EndpointID": "09edab884e498c157a092fd82ad37dcc171f197e3398ac18a1a859c9bf2eb789",
                "MacAddress": "02:42:0a:00:00:05",
                "IPv4Address": "<IP_ADDRESS>/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "257"
        },
        "Labels": {}
    }
]

Note that only the eventing-kafka is shown as being on the network -- if this step is repeated on node 1, only eventing-zookeeper is shown as being on the network
Inspect the process to verify that an IP was generated and the container is on the network:
 "NetworkSettings": {
     "Bridge": "",
     "SandboxID": "f7ab70221f7eff611a7992b2458976f0c9e6dc7edd3dacc6fec4e8edc314cc11",
     "HairpinMode": false,
     "LinkLocalIPv6Address": "",
     "LinkLocalIPv6PrefixLen": 0,
     "Ports": {
         "9092/tcp": null
     },
     "SandboxKey": "/var/run/docker/netns/f7ab70221f7e",
     "SecondaryIPAddresses": null,
     "SecondaryIPv6Addresses": null,
     "EndpointID": "",
     "Gateway": "",
     "GlobalIPv6Address": "",
     "GlobalIPv6PrefixLen": 0,
     "IPAddress": "",
     "IPPrefixLen": 0,
     "IPv6Gateway": "",
     "MacAddress": "",
     "Networks": {
         "example": {
             "IPAMConfig": {
                 "IPv4Address": "<IP_ADDRESS>"
             },
             "Links": null,
             "Aliases": [
                 "998064bfb219"
             ],
             "NetworkID": "9ysv3eb2c5js6pjkp4h06na5l",
             "EndpointID": "8bd03f75fc5fb256741f87f3ee23296564e2590892ec04206a15063d33de0d28",
             "Gateway": "",
             "IPAddress": "<IP_ADDRESS>",
             "IPPrefixLen": 24,
             "IPv6Gateway": "",
             "GlobalIPv6Address": "",
             "GlobalIPv6PrefixLen": 0,
             "MacAddress": "02:42:0a:00:00:05"

8 Attempt to telnet from inside eventing-kafka to eventing-zookeeper
docker exec -it $ps_id bash

apt-get update

apt-get install telnet

telnet eventing-zookeeper 2181

Describe the results you received:
telnet: could not resolve eventing-zookeeper/2181: Name or service not known
Describe the results you expected:
eventing-zookeeper should be accessible from eventing-kafka, even if they are not on the same machine. Both containers should be visible on a network inspect from both machines.
Additional information you deem important (e.g. issue happens only occasionally):

Are these listen-addrs reachable from the other host?

Yes, they all contain definitions for $vm_name.sseng.local in their host files. Pings between the hosts are successful.
[DkrAdmin@vm-devdocke1-sseng-local ~]$ ping VM-DevDockMgr.sseng.local
PING VM-DevDockMgr.sseng.local (<IP_ADDRESS>) 56(84) bytes of data.
64 bytes from VM-DevDockMgr.sseng.local (<IP_ADDRESS>): icmp_seq=1 ttl=64 time=0.573 ms
64 bytes from VM-DevDockMgr.sseng.local (<IP_ADDRESS>): icmp_seq=2 ttl=64 time=0.907 ms
64 bytes from VM-DevDockMgr.sseng.local (<IP_ADDRESS>): icmp_seq=3 ttl=64 time=0.326 ms
64 bytes from VM-DevDockMgr.sseng.local (<IP_ADDRESS>): icmp_seq=4 ttl=64 time=0.692 ms
64 bytes from VM-DevDockMgr.sseng.local (<IP_ADDRESS>): icmp_seq=5 ttl=64 time=0.583 ms
♥
--- VM-DevDockMgr.sseng.local ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms

It looks like using the latest master build has resolved my issues. I can confirm tomorrow.

@nostrebor Awesome! Thanks for testing.

I'll go ahead and close this issue, but please let us know if it's resolved indeed!


docs: add comments and default value for configuration

Summary

Add comments and the default value for configuration

Checklist

[ ] Does your PR title have the correct title format?
Does your PR have a breaking change?:

Thanks Alyssa for adding in-editor comments. This is really useful.
We can also add dev doc links to each configuration for users need more details like this.

Added in this commit.


No pose in odom message (and no odom <-> base link transform)

The odom message being published by this wrapper only contains Twist messages, whilst the archived repo (https://github.com/RoverRobotics/roverrobotics_stack/blob/b7538bae413b1c4f0332006f3e6612ea6657bebe/rr_rover_zero_v2_driver/src/rr_rover_zero_v2_driver.cpp#L253-L330) used to publish pose and twist (and there even seemed to be code to broadcast the odom <-> base_link transform). Is there any reason why this is not present in this repo and is this coming any time soon?

Hello, apologies for the delay in response. We are aware of the lack of the pose publishing in the current iteration. Most of our customers use robot_localization and fuse the encoder data with an IMU, so we only wanted the pose to be published once. We are looking into solutions to access pose for customers that don't want to use that, as well as assessing other quality of life features. Likely sometime in the first half of 2022.

Thanks for the response. If we wanted to calculate pose ourselves, what is the recommended way of getting odometry values other than the odometry topic? Are raw encoder ticks available or a JointState message?

We recommend using the odometry topic but only using the twist value, and using that as input into two kalman filters, one for encoders + IMU, and the second for encoders + IMU + global localization from your slam.

Also if you don't have an IMU we still recommend using two kalman filters. One for only encoders, and the second for encoders + slam. The first kalman filter that only using encoders will result in an odometry with pose.

Thanks @shoemakerlevy9 , that is what we ended up doing - just using 1 KF for the encoders to get pose. This solution works for now so feel free to close this issue.


keyboard not working on kindle fire hdx

Version of Boardwalk: 1.0.7 beta 2
Version of Android: 4.4.4
Device model: kindle fire hdx 7
Description of the problem: When I press open keyboard on boardwalk it just bugs out

In what way does it bug out? Crashes? Freezes? Graphical glitches?

The keyboard freezes and then boardwalk gets unstable and I have to close it or turn off the tablet


Are we able to improve the details provided by the WorkloadClusterCriticalPodNotRunningAWS alert?

Are we able to improve the details provided by the WorkloadClusterCriticalPodNotRunningAWS alert? (It'd be good to know what is not running)

Description now contains the pod name


298 e2e tests

Fixes: https://github.com/commercetools/commercetools-adyen-integration/issues/298

@lojzatran

It's a huge increase in time for sure, but we're testing lots of payment methods and different credit cards..

Actually do we need to test all banks? I don't think so. What is the idea here?

@lojzatran

It's a huge increase in time for sure, but we're testing lots of payment methods and different credit cards..

Actually do we need to test all banks? I don't think so. What is the idea here?

The idea is to make sure that the banks work the same. Actually when testing it, I discovered that they were not working without explicitly asking Adyen to activate :) So we know some new things here and now we can tell our customers that we have these banks tested and working. But the flow is the same for all of them in the end.

@lojzatran

It's a huge increase in time for sure, but we're testing lots of payment methods and different credit cards..

Actually do we need to test all banks? I don't think so. What is the idea here?

The idea is to make sure that the banks work the same. Actually when testing it, I discovered that they were not working without explicitly asking Adyen to activate :) So we know some new things here and now we can tell our customers that we have these banks tested and working. But the flow is the same for all of them in the end.

Solving methods activation problem in Adyen merchant account with our tests should not be the goal here, you faced this issue and you asked to activate those, in the end maybe if we test for 1 credit card instead of 12 will reduce the build time from 8 minutes to 2 minutes, which looks like an important thing for me for development experience. Here, I guess customers are not interested specifically in what we are testing every credit card type in e2e, they will just read our documentation and understand we are supporting all credit card payments, same for other payment methods also when you consider other payment types, we don't have e2e for every payment type, right?

@lojzatran

It's a huge increase in time for sure, but we're testing lots of payment methods and different credit cards..

Actually do we need to test all banks? I don't think so. What is the idea here?

The idea is to make sure that the banks work the same. Actually when testing it, I discovered that they were not working without explicitly asking Adyen to activate :) So we know some new things here and now we can tell our customers that we have these banks tested and working. But the flow is the same for all of them in the end.

Solving methods activation problem in Adyen merchant account with our tests should not be the goal here, you faced this issue and you asked to activate those, in the end maybe if we test for 1 credit card instead of 12 will reduce the build time from 8 minutes to 2 minutes, which looks like an important thing for me for development experience. Here, I guess customers are not interested specifically in what we are testing every credit card type in e2e, they will just read our documentation and understand we are supporting all credit card payments, same for other payment methods also when you consider other payment types, we don't have e2e for every payment type, right?

Our goal is to check if the credit card payment is working. The more tests we have, the better it is to argue later if sth doesn't work that it is not our fault. And now I remember that when testing credit cards, one card from that group was not working as expected as it has a bit different flow. I reported this to Adyen and they admit documentation is a bit misleading.
I would test also other methods if we have more time, but here we have a promise from Adyen that they will work out of the box.
But more tests is of course more problems - slower times and sometimes tests are timing out because response is slow. So if you don't see the benefits, I can remove them.

one card from that group was not working as expected as it has a bit different flow. I reported this to Adyen and they admit documentation is a bit misleading.

@lojzatran I guess you could keep this but the others actually are the same, so those are not important to me, just a different credit card issue and even a problem occur it will be Adyen related not extension module because we are following their contracts so we could not do much even it does not work, it will just block our build, right? I am interested to know the real value, not some exceptional cases that we can not fix, when you keep test times are slow, probably we will get bored to wait to get results, in pay one example I am bored to wait for tests, also it's different because pay one has lots of flows so it's not doing the same thing again and again. What do you think now about to drop those tests and only keep the critical ones? (same for 3Ds methods, maybe one bank might be enough).

one card from that group was not working as expected as it has a bit different flow. I reported this to Adyen and they admit documentation is a bit misleading.

@lojzatran I guess you could keep this but the others actually are the same, so those are not important to me, just a different credit card provider/bank and even a problem occur it will be Adyen related not extension module because we are following their contracts so we could not do much even it does not work, it will just block our build, right? I am interested to know the real value, not some exceptional cases that we can not fix when you keep test times slow without a doubt we will get bored to wait to get results, in pay one example I am bored to wait for tests to finish in 15 minute, also it's different because pay one has lots of flows so it's not doing the same thing again and again. What do you think now about to drop those tests and only keep the critical ones? (same for 3Ds methods, maybe one bank might be enough).

Yeah let's do it. It also makes developer's life easier :D https://github.com/commercetools/commercetools-adyen-integration/pull/321/commits/385546b67eb2a64afe8a39ffdaed38fc1321b6f0


feat(elements,ui): Add support for loading actions

Description
This PR adds support for actions, which are smaller states within a step. This allows us to disambiguate loading states within a single step. This PR adds support for the passkey and submit actions within start, but also lays the foundation for future actions within other steps. This allows us to show the correct busy state on the related UI element.
Checklist

[ ] npm test runs as expected.
[ ] npm run build runs as expected.
[ ] (If applicable) JSDoc comments have been added or updated for any package exports
[ ] (If applicable) Documentation has been updated

Type of change

[ ] 🐛 Bug fix
[x] 🌟 New feature
[ ] 🔨 Breaking change
[ ] 📖 Refactoring / dependency upgrade / documentation
[ ] other:

https://github.com/clerk/javascript/pull/3964


Implement Clone for CommandQueue

I usually use .except() instead of .unwrap() in my code, but I wanted to match the current style.

Thanks Volker.
You raise a good point about unwrap vs expect. I shall raise a new issue about error handling and investigate it further.


plugin unsupported: com.hutchind.cordova.plugins.streamingmedia @ 0.1.4

Adobe's Phonegap Build service doesn't like the latest version:
"plugin unsupported: com.hutchind.cordova.plugins.streamingmedia @ 0.1.4"
Not specifying a version provides 0.1.3 with no error.

Please try with the latest version.


update structured logging

This automatically adds additional fields like reconcile_id etc.. from the controller context.
22:57:17Z    INFO    Controllers.Horizon     Reconciling Service     {"controller": "horizon", "controllerGroup": "horizo n.openstack.org", "controllerKind": "Horizon", "Horizon": {"name":"horizon","namespace":"openstack"}, "namespace": "openstack", "nam e": "horizon", "reconcileID": "322b0ef1-491c-423d-bf14-ca0b694a861c"}
*by using per reconcile function respective logger objects, the intention is to lay the ground for parallel reconciliation in future and avoid race conditions as ctx objects are reconcile run specific.

So are we good here ?

@dprince @abays hi can you please review ?

/ok-to-test


Packaging

Create setup.py, init.py, manifest.ini and scripts for versioning, etc
See http://the-hitchhikers-guide-to-packaging.readthedocs.io/en/latest/quickstart.html
Also consider name-change, bust?

https://github.com/kennethreitz/setup.py/blob/master/setup.py


Move Cluster-specific code to rpc.cluster package

What is the goal of this PR?
We had Cluster-specific code living in the same classes as Grakn Core code. We made a conscious decision to view Cluster features as an extension of Core features, as the Cluster product is an extension of the Core product. To reflect this shift in mentality we created a new package named rpc.cluster. All Cluster functionality (besides that which cannot be easily moved, such as error messages) now lives there.
What are the changes implemented in this PR?

Move Cluster-specific code to rpc.cluster package

The changes look good!
I have two questions:

For consistency, should we call the classes under cluster rpc/RPCClientCluster.java, rpc/RPCTransactionCluster.java, and so on?
With regards to the RPC classes, we prefix it with RPC (eg., RPCTransaction) whereas in Grakn Core we put it as a suffix (eg., TransactionRPC). I think we should make it more consistent and pick one style, since there's no reason not to. What do you think?

The changes look good!
I have two questions:

For consistency, should we call the classes under cluster rpc/RPCClientCluster.java, rpc/RPCTransactionCluster.java, and so on?
With regards to the RPC classes, we prefix it with RPC (eg., RPCTransaction) whereas in Grakn Core we put it as a suffix (eg., TransactionRPC). I think we should make it more consistent and pick one style, since there's no reason not to. What do you think?

1. For consistency, should we call the classes under cluster `rpc/RPCClientCluster.java`, `rpc/RPCTransactionCluster.java`, and so on?

We could do that. I took out the word RPC because the names were getting a bit long. RPCClusterDatabaseManager ...

2. With regards to the RPC classes, we prefix it with `RPC` (eg., `RPCTransaction`) whereas in Grakn Core we put it as a suffix (eg., `TransactionRPC`). I think we should make it more consistent and pick one style, since there's no reason not to. What do you think?

That's because RPCTransaction and TransactionRPC are different things.
An RPCTransaction is an implementation of the Grakn.Transaction interface using RPC. A TransactionRPC is a container object that holds a RocksTransaction and handles RPC calls within that transaction.
But I also feel the naming on the server side isn't ideal.

1. For consistency, should we call the classes under cluster `rpc/RPCClientCluster.java`, `rpc/RPCTransactionCluster.java`, and so on?

We could do that. I took out the word RPC because the names were getting a bit long. RPCClusterDatabaseManager ...

2. With regards to the RPC classes, we prefix it with `RPC` (eg., `RPCTransaction`) whereas in Grakn Core we put it as a suffix (eg., `TransactionRPC`). I think we should make it more consistent and pick one style, since there's no reason not to. What do you think?

That's because RPCTransaction and TransactionRPC are different things.
An RPCTransaction is an implementation of the Grakn.Transaction interface using RPC. A TransactionRPC is a container object that holds a RocksTransaction and handles RPC calls within that transaction.
But I also feel the naming on the server side isn't ideal.

@alexjpwalker

We could do that. I took out the word RPC because the names were getting a bit long. RPCClusterDatabaseManager ...

Yeah, I see that too. However, let's keep the long name and favour consistency especially in the early days of the new architecture. I think once we've got a more established architecture across repositories we can start making naming tweaks for brevity.

That's because RPCTransaction and TransactionRPC are different things.
An RPCTransaction is an implementation of the Grakn.Transaction interface using RPC. A TransactionRPC is a container object that holds a RocksTransaction and handles RPC calls within that transaction.
But I also feel the naming on the server side isn't ideal.

Okay got it. Let's leave it for now and later we'll discuss it with Grakn Core people.

@alexjpwalker

We could do that. I took out the word RPC because the names were getting a bit long. RPCClusterDatabaseManager ...

Yeah, I see that too. However, let's keep the long name and favour consistency especially in the early days of the new architecture. I think once we've got a more established architecture across repositories we can start making naming tweaks for brevity.

That's because RPCTransaction and TransactionRPC are different things.
An RPCTransaction is an implementation of the Grakn.Transaction interface using RPC. A TransactionRPC is a container object that holds a RocksTransaction and handles RPC calls within that transaction.
But I also feel the naming on the server side isn't ideal.

Okay got it. Let's leave it for now and later we'll discuss it with Grakn Core people.


fix(embedded): add row level security extra cache keys for guest users

SUMMARY
Cache keys from guest token row level security clauses were not being evaluated, causing data to leak across sessions via cache.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS
In one session, create/use an guest_token without a rls clause, and a session with a rls clause with an additional cache key e.g.:
   { "clause": "publisher = '{{cache_key_wrapper('Nintendo')}}'" }
 ]

The same dashboard should have independent cache for each session, and the one with the rls clause should be unable to see unfiltered data.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

@michael-s-molina know anyone who would be well qualified to review this. I'm trying not to throw as much at @dpgaspar or @villebro though here I go mentioning them anyway ;)
Closing/reopening to kick-start CI.

@michael-s-molina know anyone who would be well qualified to review this. I'm trying not to throw as much at @dpgaspar or @villebro though here I go mentioning them anyway ;)

@rusackas We don't use embedded and RLS at Airbnb so you probably pinged the wrong person 😄


Sklearn version conflict when I use hypergbm on kaggle

System information

OS Platform and Distribution (e.g., CentOS 7.6): Linux version 5.4.144+ (builder@27db0e08a636)
Python version:
HyperGBM version: 0.2.3
Other Python packages(run pip list):

Describe the current behavior
I wanna use hypergbm on kaggle, but i got error, Looks like a version conflict
Describe the expected behavior
Just use !pip3 install -U hypergbm then i can use hypergbm on kaggle.
Standalone code to reproduce the issue
!pip3 install -U hypergbm
import hypergbm

Are you willing to submit PR?(Yes/No)
I notice here.

Attempting uninstall: scikit-learn
Found existing installation: scikit-learn 0.23.2
Uninstalling scikit-learn-0.23.2:
Successfully uninstalled scikit-learn-0.23.2

If add !pip3 install -U scikit-learn==0.23.2, it will be okay.

Installing collected packages: scikit-learn
Attempting uninstall: scikit-learn
Found existing installation: scikit-learn 1.0
Uninstalling scikit-learn-1.0:
Successfully uninstalled scikit-learn-1.0

Other info / logs

Collecting hypergbm
Downloading hypergbm-0.2.3-py3-none-any.whl (2.9 MB)
|████████████████████████████████| 2.9 MB 809 kB/s eta 0:00:01
Requirement already satisfied: plotly in /opt/conda/lib/python3.7/site-packages (from hypergbm) (5.3.1)
Requirement already satisfied: catboost>=0.26 in /opt/conda/lib/python3.7/site-packages (from hypergbm) (0.26.1)
Requirement already satisfied: distributed in /opt/conda/lib/python3.7/site-packages (from hypergbm) (2021.9.0)
Requirement already satisfied: dask in /opt/conda/lib/python3.7/site-packages (from hypergbm) (2021.9.0)
Collecting hypernets==0.2.3
Downloading hypernets-0.2.3-py3-none-any.whl (3.2 MB)
|████████████████████████████████| 3.2 MB 54.2 MB/s eta 0:00:01
Collecting dask-ml
Downloading dask_ml-1.9.0-py3-none-any.whl (143 kB)
|████████████████████████████████| 143 kB 64.3 MB/s eta 0:00:01
Requirement already satisfied: xgboost>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from hypergbm) (1.4.2)
Requirement already satisfied: imbalanced-learn>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from hypergbm) (0.8.0)
Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from hypergbm) (4.62.1)
Requirement already satisfied: fsspec>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from hypergbm) (2021.8.1)
Requirement already satisfied: lightgbm>=3.2.0 in /opt/conda/lib/python3.7/site-packages (from hypergbm) (3.2.1)
Requirement already satisfied: scikit-learn>=0.22.1 in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (0.23.2)
Requirement already satisfied: numpy>=1.16.5 in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (1.19.5)
Requirement already satisfied: pandas>=0.25.3 in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (1.3.2)
Collecting seaborn==0.11.0
Downloading seaborn-0.11.0-py3-none-any.whl (283 kB)
|████████████████████████████████| 283 kB 64.3 MB/s eta 0:00:01
Requirement already satisfied: pyarrow in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (5.0.0)
Requirement already satisfied: scipy in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (1.7.1)
Requirement already satisfied: featuretools in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (0.27.1)
Requirement already satisfied: ipython in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (7.26.0)
Collecting python-geohash
Downloading python-geohash-0.8.5.tar.gz (17 kB)
Requirement already satisfied: traitlets in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (5.0.5)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from hypernets==0.2.3->hypergbm) (1.15.0)
Requirement already satisfied: matplotlib>=2.2 in /opt/conda/lib/python3.7/site-packages (from seaborn==0.11.0->hypernets==0.2.3->hypergbm) (3.4.3)
Requirement already satisfied: graphviz in /opt/conda/lib/python3.7/site-packages (from catboost>=0.26->hypergbm) (0.8.4)
Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.7/site-packages (from imbalanced-learn>=0.7.0->hypergbm) (1.0.1)
Collecting scikit-learn>=0.22.1
Downloading scikit_learn-1.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (23.1 MB)
|████████████████████████████████| 23.1 MB 52.6 MB/s eta 0:00:01
Requirement already satisfied: wheel in /opt/conda/lib/python3.7/site-packages (from lightgbm>=3.2.0->hypergbm) (0.37.0)
Requirement already satisfied: python-dateutil>=2.7 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=2.2->seaborn==0.11.0->hypernets==0.2.3->hypergbm) (2.8.0)
Requirement already satisfied: pillow>=6.2.0 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=2.2->seaborn==0.11.0->hypernets==0.2.3->hypergbm) (8.2.0)
Requirement already satisfied: pyparsing>=2.2.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=2.2->seaborn==0.11.0->hypernets==0.2.3->hypergbm) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=2.2->seaborn==0.11.0->hypernets==0.2.3->hypergbm) (0.10.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.7/site-packages (from matplotlib>=2.2->seaborn==0.11.0->hypernets==0.2.3->hypergbm) (1.3.1)
Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas>=0.25.3->hypernets==0.2.3->hypergbm) (2021.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn>=0.22.1->hypernets==0.2.3->hypergbm) (2.2.0)
Requirement already satisfied: toolz>=0.8.2 in /opt/conda/lib/python3.7/site-packages (from dask->hypergbm) (0.11.1)
Requirement already satisfied: cloudpickle>=1.1.1 in /opt/conda/lib/python3.7/site-packages (from dask->hypergbm) (1.6.0)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from dask->hypergbm) (21.0)
Requirement already satisfied: partd>=0.3.10 in /opt/conda/lib/python3.7/site-packages (from dask->hypergbm) (1.2.0)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from dask->hypergbm) (5.4.1)
Requirement already satisfied: locket in /opt/conda/lib/python3.7/site-packages (from partd>=0.3.10->dask->hypergbm) (0.2.1)
Collecting dask-glm>=0.2.0
Downloading dask_glm-0.2.0-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: multipledispatch>=0.4.9 in /opt/conda/lib/python3.7/site-packages (from dask-ml->hypergbm) (0.6.0)
Requirement already satisfied: numba>=0.51.0 in /opt/conda/lib/python3.7/site-packages (from dask-ml->hypergbm) (0.53.1)
Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (2.4.0)
Requirement already satisfied: psutil>=5.0 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (5.8.0)
Requirement already satisfied: tornado>=5 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (6.1)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (57.4.0)
Requirement already satisfied: tblib>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (1.7.0)
Requirement already satisfied: msgpack>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (1.0.2)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (3.0.1)
Requirement already satisfied: click>=6.6 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (8.0.1)
Requirement already satisfied: zict>=0.1.3 in /opt/conda/lib/python3.7/site-packages (from distributed->hypergbm) (2.0.0)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from click>=6.6->distributed->hypergbm) (3.4.0)
Requirement already satisfied: llvmlite<0.37,>=0.36.0rc1 in /opt/conda/lib/python3.7/site-packages (from numba>=0.51.0->dask-ml->hypergbm) (0.36.0)
Requirement already satisfied: heapdict in /opt/conda/lib/python3.7/site-packages (from zict>=0.1.3->distributed->hypergbm) (1.0.1)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->click>=6.6->distributed->hypergbm) (<IP_ADDRESS>)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->click>=6.6->distributed->hypergbm) (3.5.0)
Requirement already satisfied: backcall in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (0.2.0)
Requirement already satisfied: decorator in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (5.0.9)
Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (4.8.0)
Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (3.0.19)
Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (0.1.2)
Requirement already satisfied: pygments in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (2.10.0)
Requirement already satisfied: pickleshare in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (0.7.5)
Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.7/site-packages (from ipython->hypernets==0.2.3->hypergbm) (0.18.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.7/site-packages (from jedi>=0.16->ipython->hypernets==0.2.3->hypergbm) (0.8.2)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.7/site-packages (from pexpect>4.3->ipython->hypernets==0.2.3->hypergbm) (0.7.0)
Requirement already satisfied: wcwidth in /opt/conda/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython->hypernets==0.2.3->hypergbm) (0.2.5)
Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.7/site-packages (from traitlets->hypernets==0.2.3->hypergbm) (0.2.0)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.7/site-packages (from jinja2->distributed->hypergbm) (2.0.1)
Requirement already satisfied: tenacity>=6.2.0 in /opt/conda/lib/python3.7/site-packages (from plotly->hypergbm) (8.0.1)
Building wheels for collected packages: python-geohash
Building wheel for python-geohash (setup.py) ... done
Created wheel for python-geohash: filename=python_geohash-0.8.5-cp37-cp37m-linux_x86_64.whl size=50692 sha256=0e0dc40b118fd1a5d26f093ec91dfc5cfbe771e34e3f469b9a788f8100ba7926
Stored in directory: /root/.cache/pip/wheels/ea/62/7a/e8b943f1d8025cd93a93928a162319e56843301c8c06610ffe
Successfully built python-geohash
Installing collected packages: scikit-learn, seaborn, python-geohash, dask-glm, hypernets, dask-ml, hypergbm
Attempting uninstall: scikit-learn
Found existing installation: scikit-learn 0.23.2
Uninstalling scikit-learn-0.23.2:
Successfully uninstalled scikit-learn-0.23.2
Attempting uninstall: seaborn
Found existing installation: seaborn 0.11.2
Uninstalling seaborn-0.11.2:
Successfully uninstalled seaborn-0.11.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pdpbox 0.2.1 requires matplotlib==3.1.1, but you have matplotlib 3.4.3 which is incompatible.
hypertools 0.7.0 requires scikit-learn!=0.22,<0.24,>=0.19.1, but you have scikit-learn 1.0 which is incompatible.
Successfully installed dask-glm-0.2.0 dask-ml-1.9.0 hypergbm-0.2.3 hypernets-0.2.3 python-geohash-0.8.5 scikit-learn-1.0 seaborn-0.11.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

AttributeError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in dep_map(self)
3015 try:
-> 3016 return self.__dep_map
3017 except AttributeError:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in getattr(self, attr)
2812 if attr.startswith(''):
-> 2813 raise AttributeError(attr)
2814 return getattr(self._provider, attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _parsed_pkg_info(self)
3006 try:
-> 3007 return self.pkg_info
3008 except AttributeError:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in getattr(self, attr)
2812 if attr.startswith(''):
-> 2813 raise AttributeError(attr)
2814 return getattr(self._provider, attr)
AttributeError: _pkg_info
During handling of the above exception, another exception occurred:
FileNotFoundError Traceback (most recent call last)
/tmp/ipykernel_43/4208425762.py in
----> 1 import hypergbm
/opt/conda/lib/python3.7/site-packages/hypergbm/init.py in
4
5 """
----> 6 from hypernets.experiment import CompeteExperiment
7 from .hyper_gbm import HyperGBM, HyperGBMEstimator, HyperGBMExplainer, HyperEstimator, HyperModel
8 from .experiment import make_experiment
/opt/conda/lib/python3.7/site-packages/hypernets/experiment/init.py in
7 from ._experiment import Experiment, ExperimentCallback
8 from .general import GeneralExperiment
----> 9 from .compete import CompeteExperiment, SteppedExperiment, StepNames
10 from ._callback import ConsoleCallback, SimpleNotebookCallback
11 from ._maker import make_experiment
/opt/conda/lib/python3.7/site-packages/hypernets/experiment/compete.py in
19 from hypernets.core import set_random_state
20 from hypernets.experiment import Experiment
---> 21 from hypernets.tabular import dask_ex as dex, column_selector as cs
22 from hypernets.tabular import drift_detection as dd, feature_importance as fi, pseudo_labeling as pl
23 from hypernets.tabular.cache import cache
/opt/conda/lib/python3.7/site-packages/hypernets/tabular/dask_ex/init.py in
18
19 try:
---> 20 import dask_ml.preprocessing as dm_pre
21 import dask_ml.model_selection as dm_sel
22
/opt/conda/lib/python3.7/site-packages/dask_ml/init.py in
7
8 try:
----> 9 version = get_distribution(name).version
10 all.append("version")
11 except DistributionNotFound:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in get_distribution(dist)
464 dist = Requirement.parse(dist)
465 if isinstance(dist, Requirement):
--> 466 dist = get_provider(dist)
467 if not isinstance(dist, Distribution):
468 raise TypeError("Expected string, Requirement, or Distribution", dist)
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in get_provider(moduleOrReq)
340 """Return an IResourceProvider for the named module or requirement"""
341 if isinstance(moduleOrReq, Requirement):
--> 342 return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
343 try:
344 module = sys.modules[moduleOrReq]
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in require(self, *requirements)
884 included, even if they were already activated in this working set.
885 """
--> 886 needed = self.resolve(parse_requirements(requirements))
887
888 for dist in needed:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
778
779 # push the new requirements onto the stack
--> 780 new_requirements = dist.requires(req.extras)[::-1]
781 requirements.extend(new_requirements)
782
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in requires(self, extras)
2732 def requires(self, extras=()):
2733 """List of Requirements needed for this distro if extras are used"""
-> 2734 dm = self._dep_map
2735 deps = []
2736 deps.extend(dm.get(None, ()))
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _dep_map(self)
3016 return self.__dep_map
3017 except AttributeError:
-> 3018 self.__dep_map = self._compute_dependencies()
3019 return self.__dep_map
3020
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _compute_dependencies(self)
3025 reqs = []
3026 # Including any condition expressions
-> 3027 for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
3028 reqs.extend(parse_requirements(req))
3029
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _parsed_pkg_info(self)
3007 return self._pkg_info
3008 except AttributeError:
-> 3009 metadata = self.get_metadata(self.PKG_INFO)
3010 self._pkg_info = email.parser.Parser().parsestr(metadata)
3011 return self._pkg_info
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in get_metadata(self, name)
1405 return ""
1406 path = self._get_metadata_path(name)
-> 1407 value = self._get(path)
1408 try:
1409 return value.decode('utf-8')
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _get(self, path)
1609
1610 def _get(self, path):
-> 1611 with open(path, 'rb') as stream:
1612 return stream.read()
1613
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.7/site-packages/scikit_learn-0.23.2.dist-info/METADATA'

thanks, And i  give a try as you said,  i got another error, the error is like :

AttributeError                            Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _dep_map(self)
3015         try:
-> 3016             return self.__dep_map
3017         except AttributeError:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in getattr(self, attr)
2812         if attr.startswith('_'):
-> 2813             raise AttributeError(attr)
2814         return getattr(self._provider, attr)
AttributeError: _DistInfoDistribution__dep_map
During handling of the above exception, another exception occurred:
AttributeError                            Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _parsed_pkg_info(self)
3006         try:
-> 3007             return self._pkg_info
3008         except AttributeError:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in getattr(self, attr)
2812         if attr.startswith('_'):
-> 2813             raise AttributeError(attr)
2814         return getattr(self._provider, attr)
AttributeError: _pkg_info
During handling of the above exception, another exception occurred:
FileNotFoundError                         Traceback (most recent call last)
/tmp/ipykernel_69/4208425762.py in
----> 1 import hypergbm
/opt/conda/lib/python3.7/site-packages/hypergbm/init.py in
4
5 """
----> 6 from hypernets.experiment import CompeteExperiment
7 from .hyper_gbm import HyperGBM, HyperGBMEstimator, HyperGBMExplainer, HyperEstimator, HyperModel
8 from .experiment import make_experiment
/opt/conda/lib/python3.7/site-packages/hypernets/experiment/init.py in
7 from ._experiment import Experiment, ExperimentCallback
8 from .general import GeneralExperiment
----> 9 from .compete import CompeteExperiment, SteppedExperiment, StepNames
10 from ._callback import ConsoleCallback, SimpleNotebookCallback
11 from ._maker import make_experiment
/opt/conda/lib/python3.7/site-packages/hypernets/experiment/compete.py in
19 from hypernets.core import set_random_state
20 from hypernets.experiment import Experiment
---> 21 from hypernets.tabular import dask_ex as dex, column_selector as cs
22 from hypernets.tabular import drift_detection as dd, feature_importance as fi, pseudo_labeling as pl
23 from hypernets.tabular.cache import cache
/opt/conda/lib/python3.7/site-packages/hypernets/tabular/dask_ex/init.py in
18
19 try:
---> 20     import dask_ml.preprocessing as dm_pre
21     import dask_ml.model_selection as dm_sel
22
/opt/conda/lib/python3.7/site-packages/dask_ml/init.py in
7
8 try:
----> 9     version = get_distribution(name).version
10     all.append("version")
11 except DistributionNotFound:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in get_distribution(dist)
464         dist = Requirement.parse(dist)
465     if isinstance(dist, Requirement):
--> 466         dist = get_provider(dist)
467     if not isinstance(dist, Distribution):
468         raise TypeError("Expected string, Requirement, or Distribution", dist)
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in get_provider(moduleOrReq)
340     """Return an IResourceProvider for the named module or requirement"""
341     if isinstance(moduleOrReq, Requirement):
--> 342         return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
343     try:
344         module = sys.modules[moduleOrReq]
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in require(self, *requirements)
884         included, even if they were already activated in this working set.
885         """
--> 886         needed = self.resolve(parse_requirements(requirements))
887
888         for dist in needed:
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in resolve(self, requirements, env, installer, replace_conflicting, extras)
778
779             # push the new requirements onto the stack
--> 780             new_requirements = dist.requires(req.extras)[::-1]
781             requirements.extend(new_requirements)
782
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in requires(self, extras)
2732     def requires(self, extras=()):
2733         """List of Requirements needed for this distro if extras are used"""
-> 2734         dm = self._dep_map
2735         deps = []
2736         deps.extend(dm.get(None, ()))
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _dep_map(self)
3016             return self.__dep_map
3017         except AttributeError:
-> 3018             self.__dep_map = self._compute_dependencies()
3019             return self.__dep_map
3020
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _compute_dependencies(self)
3025         reqs = []
3026         # Including any condition expressions
-> 3027         for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
3028             reqs.extend(parse_requirements(req))
3029
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _parsed_pkg_info(self)
3007             return self._pkg_info
3008         except AttributeError:
-> 3009             metadata = self.get_metadata(self.PKG_INFO)
3010             self._pkg_info = email.parser.Parser().parsestr(metadata)
3011             return self._pkg_info
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in get_metadata(self, name)
1405             return ""
1406         path = self._get_metadata_path(name)
-> 1407         value = self._get(path)
1408         try:
1409             return value.decode('utf-8')
/opt/conda/lib/python3.7/site-packages/pkg_resources/init.py in _get(self, path)
1609
1610     def _get(self, path):
-> 1611         with open(path, 'rb') as stream:
1612             return stream.read()
1613
FileNotFoundError: [Errno 2] No such file or directory: '/opt/conda/lib/python3.7/site-packages/scikit_learn-0.23.2.dist-info/METADATA'

At current timestamp, kaggle python container installed scikit-learn 0.23.2 by default. To use it with HyperGBM 0.2.3, install HyperGBM with command:
pip install scikit-learn==0.23.2  hypergbm==0.2.3

To use scikit-learn 0.24.2 with HyperGBM in kaggle container at current timestamp,  install HyperGBM with command:
pip install  --user scikit-learn==0.24.2 hypergbm==0.2.3

Inspect your kaggle container and see pip install --help for more information.


Refactor callbacks

Like structs, native-to-Java callbacks look out-of-place in Java without special JVM support. Now that LWJGL 3 requires Java 8, we should rethink their design and implementation.
Current design

Callback types are abstract classes and they extend a base abstract class (Callback.<T>).
Callback types are represented by the corresponding abstract class in bindings (function parameters and struct members).
SAM conversion is enabled with factory methods in the callback classes. There's a SAM interface in each class and a public static <CallbackClass> create(SAM sam) method.
A JNI weak global reference is created that points to the callback instance and stored in the native callback object (as "user data").
We check for premature GC when dereferencing the JNI weak global reference and display an appropriate error, without crashing.
Callback instances must be freed, by calling the .free() method on the instance.

Pros

Has worked without issues so far.
Straightforward clean-up.
Users are forced to store a strong reference to the callback instance. They don't have to RTFM, they figure it out on their own when callbacks start failing after the first GC. This then makes it relatively obvious that clean-up must take place.

Cons

SAM conversion does not work with abstract classes, only with functional interfaces.
There's both a class and an interface (SAM) for each callback type.
Callback class names are arbitrary; they are not present as a type in the native API.
The user is forced to store a strong reference to the callback instance.

Recent changes

LWJGL 3 now requires Java 8. We can use static and default methods in interfaces.
libffi has been replaced by dyncall. One important difference with dyncall is that the thunk pointer and the callback object handle are guaranteed to be one and the same. This means that we only need a single pointer to reference the entire (native and Java) callback structure.

Proposed design

Each callback type is a functional interface.
Callback argument decoding can happen directly in the interface, with a default method.
The Callback class becomes a utility class that only creates and destroys callback objects.
A callback object is a plain long value (opaque handle). Callback function arguments and member fields become long.
There are method overloads that take a functional interface, create a callback object from it and pass it to the native function/struct.
Callback type instances are stored in strong JNI global references.

Note that non-capturing lambdas and method references are never GCed anyway.

The user may or may not store the callback object in Java code. In many bindings they don't have to store it, it's readily available via the API (e.g. glfwSet<Type>Callback function return the previously set callbacks).

Pros

There's a single functional interface per callback type.
One less indirection (no abstract class -> SAM delegation).
Interface names won't show up in code that uses lambdas or method references.
Callback object handles stored in user code will be simple long, not arbitrary types.
Less callback objects stored in user code.
No worries about premature GC.

Cons

Memory leaks if the user doesn't perform clean-up. This is generally of low-concern, most applications will use a low-number of callbacks without much (if any) state.
Non-obvious clean-up:

Must retrieve callback object handle using the appropriate binding API (e.g. glGetPointer(GL_DEBUG_CALLBACK_FUNCTION)), if the handle was not already stored in user code.
Must use Callback.free(handle) (vs cbInstance.free() currently).

Example 1, similar to current design:
private long keyCB; // callback handle is a long instead of an object
// ...
// callback setup, must pass lambda or method reference to the static create method
glfwSetKeyCallback(window, keyCB = GLFWKeyCallback.create((windowHandle, key, scancode, action, mods) -> {
   // ...
 }));
// cleanup
Callback.free(keyCB);

Example 2, without storing the callback handle:
// callback setup, lambda or method reference passed directly to the API
glfwSetKeyCallback(window, (windowHandle, key, scancode, action, mods) -> {
   // ...
 });
// cleanup
Callback.free(glfwSetKeyCallback(window, null));

I would prefer it be in MemoryUtil instead of Callback as its the class that already does most of the memory manipulation stuff and from what it looks like most of the Callback classes use will be internal except for the free() method. Also I would like it if the callbacks return the object instead of a pointer though this is mostly because Java works with objects and not pointers for the most part though I would be fine ether way. Though I would very much like this change as my code already does something similar to your second example though looks less nice because of the way lambdas have to be used

I would prefer it be in MemoryUtil instead of Callback as its the class that already does most of the memory manipulation stuff and from what it looks like most of the Callback classes use will be internal except for the free() method.

A fine idea, thanks!

Also I would like it if the callbacks return the object instead of a pointer though this is mostly because Java works with objects and not pointers for the most part though I would be fine ether way.

This is something I'd like to avoid:

There's no need for a wrapper object. It would also be an extra indirection in the native-to-Java callback path.
I'd like to minimize LWJGL-specific abstractions/utilities visible in user code (in the spirit of making LWJGL code look very similar to the corresponding C code). Note that in example 1 above there's only one mention of GLFWKeyCallback and no mention at all in example 2.
I've written a lot of GLFW samples and always find it annoying having to remember callback type names. Auto-complete helps in the glfwSet methods, but not when writing the fields where the callback instances are stored.

I am probably misunderstanding what you are trying to say so correct me if I get something wrong but

There's no need for a wrapper object. It would also be an extra indirection in the native-to-Java callback path.

No need for one as its fully legal to store it and return it latter as the Interface class without wrapping it in another object. You can even wrap other objects in lambda methods the same way you would wrap lambda methods in full classes.

I'd like to minimize LWJGL-specific abstractions/utilities visible in user code (in the spirit of making LWJGL code look very similar to the corresponding C code). Note that in example 1 above there's only one mention of GLFWKeyCallback and no mention at all in example 2.

The return type would simply change from a long to the callback interface. Nothing else would need to change as it would still be a Functional interface. It would also allow for the extra overload methods that take the long (though I suppose the object type methods would be the helper methods) to be removed and not seen from the user code.

I've written a lot of GLFW samples and always find it annoying having to remember callback type names. Auto-complete helps in the glfwSet methods, but not when writing the fields where the callback instances are stored.

I don't know how garbage collection works with lambda functions as I have not really looked that closely at it however you said that currently you use a weak reference to an object to store it. You could do something similar for the lambda functions if needed without the weak reference part though I don't really know why lwjgl was storing that in the first pace instead of a normal reference unless memory was a concern for some reason in which case it should have been a soft reference not a weak one.

No need for one as its fully legal to store it and return it latter as the Interface class without wrapping it in another object. You can even wrap other objects in lambda methods the same way you would wrap lambda methods in full classes.

There are two pieces of state:

The interface implementation. Can be an anonymous class instance, a lambda, or a method reference.
The native callback object (currently implemented with dyncall).

The native callback object already contains a (JNI global) reference to the interface implementation. The interface implementation does NOT know anything about the native callback object (it can't know anything if we want lambdas to work). If we want an object return value, the only option is to wrap the native callback object handle in a class. What would be the point of that?

Its supposed to return the previously set callback or null so I was suggesting a way to possibly do that (a map or something with the long value as the key and the function as the value) in object form but it should be fine ether way if nothing other than the long is needed to reset the function to what it was before if it is changed

Sorry, I'm probably not explaining things clearly. Lets try again.
The dyncall callback object (DCCallback*) is a struct. The long value you get from <CallbackInterface>.create(...) is an opaque pointer to that struct. The first member of that struct is the thunk, which is basically the function pointer passed to a native API that expects a callback pointer. Since it's the first member in the struct, the DCCallback* pointer is also the function pointer. You have both using a single long value.
The only thing missing now is the Java callback (the instance that implements the callback interface). This is also stored in the callback object struct, in the userdata field. Since the struct is opaque, dyncall has a function for retrieving the userdata field: dcbGetUserData(DCCallback*). So you can do this:
long keyCB = GLFWKeyCallback.create((windowHandle, key, scancode, action, mods) -> {
   // ...
 });

GLFWKeyCallback javaCallback = memGlobalRefToObject(dcbGetUserData(keyCB));
// or simply
GLFWKeyCallback javaCallback = Callback.get(keyCB);

I still think it would be more like java to return the actual callback (return memGlobalRefToObject(dcbGetUserData(keyCB)) instead of just the long value) though as long as its easy enough to reach the java object it would be fine and getting the long would probably also save some calls to recreate or get the object pointer later if it is reset as the callback.
Mostly I was thinking of libraries that may chose the way of simply overriding the current callback with their own and calling the former callback after they have finished processing the data. Not the best way of doing things but far easier than some other ways. Though again as long as the java callback can be easily reached its fine.

I have no issue with opaque pointers. It'd be really straightforward to make wrapper "create" that call Callback.get(cb) inside if needed.
Also not a lot of code would actually call these "GLFWKeyCallback.create" anyway, so making it slightly more verbose for the cases where you do want to retrieve the callback object shouldn't have much impact, if at all.

After implementing the proposed design above and changing all sample code to use it, I ended up not liking it very much. Specifically, I didn't like having to write memFreeCallback(cbHandle); in clean-up code. After a few more experiments, the final design is a bit weird, but basically what @ShadowLordAlpha requested. If someone reviews the current implementation and has better ideas, please don't hesitate to post here or open a new issue.
Final design
Example 1:
private GLFWKeyCallback keyCB;
// callback setup
glfwSetKeyCallback(window, keyCB = GLFWKeyCallback.create((windowHnd, key, scancode, action, mods) -> {
   // ...
 }));
// cleanup
keyCB.free();

Example 2, without storing the callback handle:
// callback setup
glfwSetKeyCallback(window, (windowHnd, key, scancode, action, mods) -> {
   // ...
 });
// cleanup
glfwSetKeyCallback(window, null).free();

Example 3, easy GLFW clean-up:
glfwFreeCallbacks(window); // in the org.lwjgl.glfw.Callbacks class
glfwDestroyWindow(window);

glfwTerminate();
glfwSetErrorCallback(null).free();

Important:
GLFWKeyCallback keyCB = (windowHnd, key, scancode, action, mods) -> {
   // ...
};
keyCB.free(); // the lambda is stateless, will throw UnsupportedOperationException

GLFWKeyCallback keyCB = GLFWKeyCallback.create((windowHnd, key, scancode, action, mods) -> {
   // ...
});
keyCB.free(); // works

The change you made makes it impossible to implement multiple callbacks now, since the shared invoke(long) default conflicts among the different callbacks. Is it just not advised to directly implement more than one callback?

@kenzierocks Could you share an example please? You're talking about multiple callbacks with compatible function signatures, right?

The previous solution was too confusing and error prone. After further discussion a new approach has been implemented that is compatible with existing code, supports all code styles and has no clean-up surprises.
Final design
Example 1 (anonymous class):
private GLFWKeyCallback keyCB;
// callback setup
glfwSetKeyCallback(window, keyCB = new GLFWKeyCallback() {
	@Override
	public void invoke(long window, int key, int scancode, int action, int mods) {
		// ...
	}
});
// cleanup
keyCB.free();

Example 2 (indirect lambda):
private GLFWKeyCallback keyCB;
// callback setup
glfwSetKeyCallback(window, keyCB = GLFWKeyCallback.create((windowHnd, key, scancode, action, mods) -> {
	// ...
}));
// cleanup
keyCB.free();

Example 3 (direct lambda):
// callback setup
glfwSetKeyCallback(window, (windowHnd, key, scancode, action, mods) -> {
	// ...
});
// cleanup
glfwSetKeyCallback(window, null).free();

Implementation

Each callback type now generates an interface and an abstract class. For example, the GLFW callback type GLFWkeyfun generates the GLFWKeyCallbackI interface and the GLFWKeyCallback abstract class.
Input callback types (function parameters or struct setter values) are mapped to the corresponding interface.
Output callback types (function or struct getter return values) are mapped to the corresponding abstract class.
With the new class hierarchy, there is no free() method in callback interfaces, only in abstract classes. This should make it obvious that clean-up requires a stateful instance and that interfaces are there to enable input lambdas.

This is what I used to do @Spasi, extending the SAM interface. This might not be supported, would you recommend I do something like this instead?

Method references are perfect for this case. See MouseHelp with method references.
Also, don't forget to clean up your callbacks. LWJGL provides a convenient method for freeing GLFW callbacks: org.lwjgl.glfw.Callbacks.glfwFreeCallbacks(long window)

OK, I'll do that instead. Thanks!


The memory occupied by Loader cannot be released

I use datavec in tomcat.
I find that after each use of NativeImageLoader.asMatrix(mat), the memory will grow by nearly 100MB and will not be released afterwards.
DL4j: 0.9.1
Tomcat: 7.0.82
Javacv: 1.4

Oh, maybe MemoryWorkspace can work out

It does get released, just not very efficiently. This will probably get sorted out with pull #573.


MongoDB Throws "Duplicate vertices: 64 and 66" in FL-21

It seems like the coordinate pair [-80.036362, 26.77124] appears twice in the GeoJson file and Mongo doesn't like that for some reason.

Solved:
https://github.com/unitedstates/districts/pull/8


POC: Troubleshoot command

We often ask for the same info when debugging issues. It might be useful to have a command system-info that automatically collects some basic info. Users can then be told to run this command and attach its output to issue reports.
Example of the current version:
>>> qlever system-info

Command: system-info

================== General Info ==================
qlever-control: 0.5.2
OS: Linux (Arch Linux)
Arch: x86_64
Host: archmobile
RAM: 30.1 GB total, 19.5 GB available
CPU: 8 Cores, 16 Threads @ 5.13 GHz
CWD: /home/julian/code/uni/osm-live-updates-qlever
Disk space in . (/dev/nvme0n1p3 @ / is ext4): 142.16GB free / 905.34GB total
User/Group on host: uid=1000(julian) gid=1000(julian) groups=1000(julian),961(docker),1002(sudo)
=================== Qleverfile ===================
# Qleverfile for OSM Planet, use with the qlever script (pip install qlever)
#
# qlever get-data  # takes ~50 mins to download .ttl.bz2 file of ~ 300 GB
# qlever index     # takes ~12 hours and ~20 GB RAM (on an AMD Ryzen 9 5900X)
# qlever start     # takes a few seconds
#
# For the OSM data of a single country, do `qlever setup-config osm-country`
# and edit the Qleverfile to specify the country,

[data]
NAME         = osm-andorra
#DATA_URL     = https://osm2rdf.cs.uni-freiburg.de/ttl/planet.osm.ttl.bz2
DATA_URL     = https://osm2rdf.cs.uni-freiburg.de/ttl/and.osm.ttl.bz2
GET_DATA_CMD = curl --location --fail --continue-at - --remote-time --output ${NAME}.ttl.bz2 ${DATA_URL}
VERSION      = $$(date -r ${NAME}.ttl.bz2 +"%d.%m.%Y")
DESCRIPTION  = OSM Planet, data from ${DATA_URL} version ${VERSION} (complete OSM data, with GeoSPARQL predicates ogc:sfContains and ogc:sfIntersects)

[index]
INPUT_FILES     = ${data:NAME}.ttl.bz2
CAT_INPUT_FILES = lbzcat -f -n 2 ${INPUT_FILES}
STXXL_MEMORY    = 20G
SETTINGS_JSON   = { "languages-internal": [], "prefixes-external": [""], "ascii-prefixes-only": false, "num-triples-per-batch": 5000000 }
INDEX_BINARY    = /home/julian/CLionProjects/qlever/cmake-build-release/IndexBuilderMain

[server]
PORT                        = 7007
ACCESS_TOKEN                = ${data:NAME}_oQSb1fI7z87h
MEMORY_FOR_QUERIES          = 90G
CACHE_MAX_SIZE              = 40G
CACHE_MAX_SIZE_SINGLE_ENTRY = 30G
TIMEOUT                     = 300s
SERVER_BINARY               = /home/julian/CLionProjects/qlever/cmake-build-release/ServerMain

[runtime]
SYSTEM = native
IMAGE  = docker.io/adfreiburg/qlever:latest

[ui]
UI_CONFIG = osm-planet

We can also pre-fill the issue description to prompt users to run and attach qlever system-info. This is independent of this issue and we can also discuss it later. It could look something like this: