--test-split and --json-filter don't work together

I've got a master dataset with lots of labels and features.
I want to manually generate a split for every label with --test-split so that I can create multiple ensembles and evaluations against it (different feature sets).
I want to use --json-filter to make sure that a split only contains labeled records - in other words, ["not",["missing", "labelX"]]. That way I can ensure that train and test really are 70%/30%.
Currently, --test-split and --json-filter don't work together - the filter is ignored.

Closing this issue. It was solved some time ago. If you experience the problem again, please reopen.


Add table aliases

Table aliases could make qualified references more succinct.
public
  some_long_table_name as sltn
    record
      ..cols

  another_table
    record
      some_col<EMAIL_ADDRESS>
This still retains search capabilities, since it's easy to search for "as sltn" to remember what it stands for in longer files.

This could also render https://github.com/kevlarr/hldr/issues/33 less important

Added via https://github.com/kevlarr/hldr/pull/42


French-Wilson script

@kmdalton what do you think? If you guys have a good implementation, a simple CLI utility to turn intensities into SFAs would be great.
I'd love to not have to use phenix.reflection_file_converter...

And I should say -- I volunteer myself for this task.

seems reasonable -- we have a French-Wilson implementation in reciprocalspaceship: https://github.com/rs-station/reciprocalspaceship/blob/cbf30c1c8438326baddf6a9898c2f5b5be3f7b43/reciprocalspaceship/algorithms/scale_merged_intensities.py#L178
We also currently use this in rs-booster as an internal IO convenience, so I agree it would make sense to have a standalone CLI if you think that would be useful: https://github.com/rs-station/rs-booster/blob/741c2bba9dbad3b0fd704bba68cfb7da41d2f1b3/rsbooster/utils/io.py#L40-L46

Yeah this makes sense to me. It should be relatively quick to implement. Let me know if you need help.


More label is displaying in multiple cells

Hi @Sheena1234 & @acro5piano,
Thank you for the fix in the latest version, but there is a one issue I got it today. Please have a look at attached image. Can you guys please fix it?

Thank you.

Hi @noamanmansuri Thanks for reporting this.
I understand the issue, but I'm too busy to fix it now. If you send a pull request, I'm happy to review and merge it.
Best Regards,


Change inline @file context cause message loss after it

Before submitting your bug report

[X] I believe this is a bug. I'll try to join the Continue Discord for questions
[X] I'm not able to find an open issue that reports the same bug
[X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info
- OS: macOS 15.0
- Continue: 0.8.52
- IDE: VSCode 1.93.1
- Model: any
- config.json:

  {
  "models": [
    {
      "model": "claude-3-5-sonnet-20240620",
      "contextLength": 200000,
      "title": "Claude 3.5 Sonnet",
      "apiKey": "*********",
      "provider": "anthropic"
    }
  ],
  "customCommands": [
    {
      "name": "test",
      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Starcoder2 3b",
    "provider": "ollama",
    "model": "starcoder2:3b"
  },
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "edit",
      "description": "Edit selected code"
    },
    {
      "name": "comment",
      "description": "Write comments for the selected code"
    },
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

Description
https://github.com/user-attachments/assets/e241ba00-def4-4221-b4d9-b9177f2dbf9d
To reproduce

Write a prompt: look @file1 and @file2, write code
Change @file1 to another file, the rest of message (and @file) will disappear

Log output
No logs in the console.

Thanks for the screen recording here @Joilence ! Very helpful.
My best guess is that there is some React state logic here that we are clearing out accidentally when you attempt to edit a context provider that comes before other providers.
Not sure how soon we'll be able to get around to fixing but if you're interested in taking a look that would be much appreciated! 😁
https://github.com/continuedev/continue/blob/main/CONTRIBUTING.md

Thanks for the reply! I am not familiar with React, hopefully this will catch someone else's attention :)

#2387

Thanks for the screen recording here @Joilence ! Very helpful.
My best guess is that there is some React state logic here that we are clearing out accidentally when you attempt to edit a context provider that comes before other providers. The logic for that is in this file: https://github.com/continuedev/continue/blob/3f8402af19ae010b60adaf893f06548f3cef877c/gui/src/components/mainInput/TipTapEditor.tsx
Not sure how soon we'll be able to get around to fixing but if you're interested in taking a look that would be much appreciated! 😁
https://github.com/continuedev/continue/blob/main/CONTRIBUTING.md

Yes I came out with a fix. Was this intentional? I believe it's a better UX if a user has a long prompt and wants to change one file at the beginning it won't remove the entire prompt.

Perfect fix. It was intentional only to remove the text right after the "@", so your solution was great


404 error for IEEE link on Python Example page

Describe the bug
When clicking the hyperlink "IEEE site" on the Python Example page, the user is taken to a 404 error.
To Reproduce
Steps to reproduce the behavior:

Go to https://www.freecodecamp.org/news/python-example/
Click on the hyperlink 'IEEE site'
You are taken to a 404 error on the IEEE website: http://grouper.ieee.org/groups/754/

Expected behavior
The user is taken to a page regarding IEEE 754 specification: Standard for Binary Floating-Point Arithmetic. Not sure if this is now the current link: http://grouper.ieee.org/groups/msc/ANSI_IEEE-Std-754-2019/background/ieee-computer.pdf
Screenshots

Desktop (please complete the following information):
OS: Mac OS X 10.14.6
Browser Chrome
Version 79.0.3945.130

Maybe use the following Wikipedia link?

https://en.wikipedia.org/wiki/IEEE_754

@DimplePatel Thanks for catching this. I've fixed this.
@RandellDawson I've used your recommended wikipedia link.


Add handoff support for issues

Ref #328.
While implementing this I was wondering though... opening a web page with Handoff would automatically mark an issue as read (as you would load the page on GitHub.com). Do we maybe want to only allow Handoff when the user has this setting enabled?
This would only make sense for the issues though - we can still support handoff for readmes, profiles etc. Not sure what would make most sense here.

Is this a two way handoff or only from App -> Web? Would be handy to quickly pick up where I was!
I also feel as a user you expect that going to the web will have slightly different functionality, I think it's fine that it reads it

Would be handy to quickly pick up where I was!

What? You were working on this?
And yes, this is only app > web.

You were working on this?

No! Sorry I meant it would be handy to pick up where I was so that if I was replying to issues on my desktop and wanted to jump in a taxi then I could quickly pick it up on my phone

Ha, was confused. It is unfortunately not possible to go from Web -> app, unless it’s your domain. As we are working with GitHub... 😔
What we could do is to have a url scheme / share extension that can open GH urls in the app?

What we can also do is Handoff from app -> app. So switching from iPad to iPhone would be easier.

App to App would be good, I also know a month or so ago I spoke to Ryan about having a custom domain as a middleman for GitHub
Long explanation short a domain could be something like githawk.com/rnystrom/GitHawk/pull/331 - If the app is installed then it will take them to it, otherwise it just redirects the user to the github.com alternative. (Universal linking)
That in combination with share extensions would be a powerful combination

Yep, the middelman route is pretty neat.

I'll take this for now and we can try it out. I honestly don't see this getting a ton of use, so we can fix the edge cases and UX if ppl start using it a lot.


no 64 bit support  ?

I am not sure if someone else has the problem but i cannot install the app.
It complaints that it is not 64 bit compatible.

It is a 32 bits application but will work in any Windows 64 bits. Which version is your Windows? Please put the exact message you get here.

Hi, i apologize.
I now tried to install it a second time to write down the exact error.
But this time it just installed and startet up the application, it works now.
I don't know why the error message occured at the first trial.

No worries! It may be a corruption during download.


[BUG] Prevents Start up of servers

So.... after 3 hours of debugging 250 mods, your mod prevented world creation in a server instance.
I did this one by one, disabling mod after mod, and it came down to Immersive Weapons. There is no log indicating why, but there is a crash error I can provide .
Latest Immersive Weapon version
1.18
Modloader 38.0.17
[20:24:49] [main/FATAL]: Failed to start the minecraft server
java.lang.RuntimeException: Missing referenced custom/removed registry entry for registry ResourceKey[minecraft:root / minecraft:worldgen/biome] named minecraft:overworld
at com.mojang.serialization.DataResult.lambda$getOrThrow$4(DataResult.java:100) ~[datafixerupper-4.0.26.jar%2329!:?]
at com.mojang.datafixers.util.Either$Right.map(Either.java:99) ~[datafixerupper-4.0.26.jar%2329!:?]
at com.mojang.serialization.DataResult.getOrThrow(DataResult.java:93) ~[datafixerupper-4.0.26.jar%2329!:?]
at net.minecraft.server.Main.main(Main.java:175) ~[server-1.18-20211130.085255-srg.jar%23265!:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at net.minecraftforge.fml.loading.targets.CommonServerLaunchHandler.lambda$launchService$0(CommonServerLaunchHandler.java:46) ~[fmlloader-1.18-38.0.17.jar%2323!:?]
at cpw.mods.modlauncher.LaunchServiceHandlerDecorator.launch(LaunchServiceHandlerDecorator.java:37) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.modlauncher.LaunchServiceHandler.launch(LaunchServiceHandler.java:53) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.modlauncher.LaunchServiceHandler.launch(LaunchServiceHandler.java:71) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.modlauncher.Launcher.run(Launcher.java:106) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.modlauncher.Launcher.main(Launcher.java:77) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.modlauncher.BootstrapLaunchConsumer.accept(BootstrapLaunchConsumer.java:26) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.modlauncher.BootstrapLaunchConsumer.accept(BootstrapLaunchConsumer.java:23) [modlauncher-9.0.7.jar%235!:?]
at cpw.mods.bootstraplauncher.BootstrapLauncher.main(BootstrapLauncher.java:90) [bootstraplauncher-0.1.17.jar:?]

I'll take a close look at the issue soon. But just from glancing at the log snippet, it seems like it's a worldgen problem. Which is odd, because I thought I had disabled all of it for the 1.18 beta releases (due to issues in the Forge API).

Found the problem. Part of the datapack for working with custom dimensions wasn't fully disabled, and so it would fail since the entries weren't being registered. I'll have a new beta release out soon.

Should be fixed as of the latest 1.18.1-beta3 release. Let me know if anything else breaks.

will try it out now, thank you.


Redesigned app launcher icon

Icon redesigned following material design guidelines and matching with app colors

I tried using this on my phone and I like the current one better. The musical note seems a bit off the center due to it's shape.
Still, thanks for the design though. Appreciated!

Ok no problem! If you want another icon, just send me it and we'll see if it will be good


Bug Report: lime test windows

Please check for duplicates, similar issues or closed issues with the same problem, as well performing simple troubleshooting steps (such as deleting the export folder if it's an issue with compiling, reinstalling Kade Engine) before submitting an issue.
If you are playing a downloaded version of the game, what operating system are you using?
Windows (x86), Windows (x86_64), Linux, or macOS? Specify below.
windows x64
What version of the game are you using? Look in the bottom left corner of the main menu.
current version
Have you identified any steps to reproduce the bug? If so, please describe them below. Use images if possible.
no
Please describe your issue. Provide extensive detail and images if possible.

Oh wow u got a different issue its not caused by the API thing atleast

yeah


How do I setup the credentials for the SDK?

I put together a test function for lambda, but the credentials are not set into the SDK.
-- Error: Missing credentials in config
Am I using the same copy of the SDK as amplify?
Had amplify set the credentials into the SDK?
I also noted that AWS.config.region is not set either.
Previously I did setup like this:
AWS.config.update({region: 'us-east-1'});
AWS.config.credentials = new AWS.Credentials({
accessKeyId: sessionStorage.getItem('accessKeyId'),
secretAccessKey: sessionStorage.getItem('secretAccessKey'),
sessionToken: sessionStorage.getItem('sessionToken'),
});
But I had expected aws-amplify to set this up for me.
import React, { Component } from 'react';
import AWS from 'aws-sdk'

var Lambda = new AWS.Lambda({region: 'us-east-1', apiVersion: '2015-03-31'});

export default class Page2 extends Component {

  build_table () {
    // create JSON object for parameters for invoking Lambda function
    var pullParams = {
      FunctionName : 'digi-dev-getEvents',
      InvocationType : 'RequestResponse',
      LogType : 'None'
    };
    // create variable to hold data returned by the Lambda function
    var events;

    Lambda.invoke(pullParams, function(error, data) {
      if (error) {
        prompt(error);
      } else {
        events = JSON.parse(data.Payload);
        var t = "
";
        events.forEach(function(r) {
          t += "
";
          var x = new Date( r["timestamp"] );
          var formatted = x.toLocaleString();
          t += "" + formatted + "	";
          t += "" + r["thing"] + "	";
          t += "" + r["type"] + "	";
          t += "
";
        });
        t += "
";
        console.log(t);
      }
    });
  }

Do I need to do something like this?
_ensureCredentials() {
    if (this._options.credentials) { return Promise.resolve(true); }

    return Auth.currentCredentials()
        .then(credentials => {
            const cred = Auth.essentialCredentials(credentials);
            logger.debug('set credentials for storage', cred);
            this._options.credentials = cred;

            return true;
        })
        .catch(err => {
            logger.warn('ensure credentials error', err)
            return false;
        });
}

I just expected amplify to supply me with a global AWS handle that always has valid credentials in it (ie refresh is taken care of).

I did a Pull Request adding Lambda support


deprecate and relocate connector related endpoints in functions REST API

Motivation
There are several connectors (sources/sinks) related admin APIs and REST endpoints that reside within functions admin API and REST endpoints which isn't necessary any more since sources and sinks having their own separate admin API and REST API now.
Also move getting a complete list of connectors (sources and sinks) for a worker from functions REST API to Worker API since that more sense.

rerun java8 tests

rerun java8 tests


GridLayout (tap) binding works in Android, partially in iOS

Environment
Provide version numbers for the following components (information can be retrieved by running tns info in your project folder or by inspecting the package.json of the project):

CLI: 5.4.0
Android Runtime: 5.4.0
iOS Runtime: 5.4.2
NativeScript-Angular: 7.2.0
Angular:7.2.0

Describe the bug
Bind a Gridlayout, 2 columns, with the tap event, using ngFor:
<GridLayout row="2" rows="*,*,*"  >
        <GridLayout [row]="icon.mainRow"  columns="*,*" *ngFor="let icon of icons.myData">
            <GridLayout (tap)=doAction(icon.linkCommand) [col]="icon.colNumber" rows="auto,auto">
                <Image row="0" [src]="icon.Image"></Image>

            </GridLayout>
        </GridLayout>
    </GridLayout>

Note: the structure of the Grid is being built dynamically by passing in row and col for each new data item.
This works great on Android, with each (tap) event firing for each element.
However, in iOS, only the second column (col='1') fires on tap.
It does not matter where I place the (tap), on the image, the label, the same result.
Works on Android, only works for second column on iOS.
Here is the other interesting thing, all other elements are bound properly on both devices.
I get the Text and Image, as well as the correct column and row for each iteration of the data to the GridLayout on both devices.
Why only the second column, on iOS only, will fire the (tap), I am at a loss.
To Reproduce
Layout grid as above, bind some sample data to it, for rows, cols, and text, image, link.
Build for both platforms.
Expected behavior
All (tap) events should fire for both columns on both platforms.

Hi @SuperdanTX,
I was able to recreate the issue in a sample project, will mark it as a bug. We will investigate it further and for now, I would suggest keeping track of the problem for more info.

Just a follow up on this issue: We upgraded the project to 6.0, same result as above.
I had seen your comment in another post to change from [col]="thing" to col="{{thing}}", but no fix there either for this purpose.
Thanks!

Hi @SuperdanTX,
Can you send us the project with the latest changes, where you have applied the workaround(changing [col]="thing" -> col="{{thing}}")?

You can refer to your sample project above.
I simply changed to the above like this, with the same failed result.
The first column wont handle the click, the second will.
<GridLayout row="{{icon.mainRow}}" columns="*,*" *ngFor="let icon of icons">         <GridLayout (tap)=doAction(icon.linkCommand) col="{{icon.colNumber}}"
Previous binding:
<GridLayout [row]="icon.mainRow" columns="*,*" *ngFor="let icon of icons">         <GridLayout (tap)=doAction(icon.linkCommand) [col]="icon.colNumber"
Neither way will fix the first column failing to handle click event.

Hello @tsonevn,
I know it has been a while with this issue, but any update 2 years later ? I tried to do [col] and col="{{ }}" for my values. Also only had label inside my ngFor but the problem is still happening :
<GridLayout row="1" columns="*, *, *" *ngFor="let item of journey | slice:0:3 ; let idx = index"> 	<GridLayout iosOverflowSafeArea="false" col="{{idx}}" class="labelList" [class.selected]="idx === journeyIdx" (tap)="onTap('typeOfJourney', item, idx)" > 	 </GridLayout> </GridLayout> <GridLayout row="2" columns="*, *" *ngFor="let item of journey | slice:3:5 ; let idx = index" iosOverflowSafeArea="false"> 		  </GridLayout>
2 different types of ngFor but none of them works, also tried with iosOverflowSafeArea but not working


descendFirst on $scope.query

Would be nice if you could choose descendFirst on initial $scope.query in controller. Useful for dates.
$scope.query = {
    order: 'name',
    descendFirst: true,
    limit: 5,
    page: 1
};

Does using a minus sign work?
$scope.query = {
  order: '-name',
  limit: 5,
  page: 1
};

Then you can put the descendFirst attribute on the column header to have it sort descending by default when the user clicks it. Let me know if this works for you.
I found a bug while testing this for you. When you hover a column header with the descend first attribute the arrow icon should point downwards. The column will still sort descending though.

Ah! You my friend are a genius, thanks! Sorry for taking up your time.

No problem! This issue lead me to discover that other bug, which should be fixed now. :+1:


rust: add v1.72.0 for zenoh-c/0.10.1-rc

This PR adds Rust 1.72.0.
As mentionned in the Zenoh-C CCI PR, it would be ideal to have Rust 1.72.0 to ensure compatibility with Zenoh-C's 0.10.1-rc release artefacts, mainly regarding plugins.

@oteffahi
I already added 1.72, but added 1.72.1 instead of 1.72.0. Is that ok?

As mentionned in the Zenoh-C CCI PR, it would be ideal to have Rust 1.72.0 to ensure compatibility with Zenoh-C's 0.10.1-rc release artefacts, mainly regarding plugins.

Is this still relevant if we are building zenoh-c from scratch instead of using prebuilt binaries? Even then, Rust stdlib and other libraries are linked statically into the library, so I think the Rust version still should not really matter?

Hello @valgur,
Rust 1.72.0 is required in order for the resulting build to be compatible with prebuilt plugins and backends that are part of the 0.10.1-rc release. Zenoh users tend to want to use those without going through the bother of compiling them from source, so we want them to be supported if possible. This requires that both zenoh and the plugins must be built with the same Rust version, which in the case of 0.10.1-rc happens to be 1.72.0.
I hope this answers your question regarding this requirement.

Makes sense, thanks! I'll downgrade the version in the Rust recipe to 1.72.0.

Actually, on second thought, I would assume that 1.72.1 is still compatible with binaries built with the .0 version. Correct me if I'm wrong.

Actually, on second thought, I would assume that 1.72.1 is still compatible with binaries built with the .0 version. Correct me if I'm wrong.

@valgur sorry for the late reply. We haven't tested this, but ideally to avoid any possible issues it's better to use the exact same version. Note that we maintain a rust-toolchain.toml file which contains the exact version to be used when compiling the project. This version is downloaded and installed automatically by Cargo if building without providing a channel (i.e using cargo build instead of cargo +stable build or cargo +1.72.1 build for example)
I understand that you may want to support the latest patch of each minor version of Rust, but projects can require a specific version through their rust-toolchain.toml file, which might not be on the latest patch. In our case this is done in order to ensure compatibility with release artefacts (plugins).


Need Assistance for Prediction from image

Dear All,
I'm not sure about prediction statement can you help me to correct it ?
X = cv2.imread('/content/download (6).png')
model.predict(X)
------------------ ERROR-------------
low:Model was constructed with shape (None, None, None, 3) for input KerasTensor(type_spec=TensorSpec(shape=(None, None, None, 3), dtype=tf.float32, name='input_1'), name='input_1', description="created by layer 'input_1'"), but it was called on an input with incompatible shape (32, 834, 3).
ValueError: Input 0 of layer stem_conv is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: (32, 834, 3)

Probably your model requires a batch to perform prediction:
X = cv2.imread('/content/download (6).png')
X = np.expand_dims(X, 0)
model.predict(X)


Bootstrap pod: Failure no such index [.opendistro_security]

Hi,
I am currently trying to switch to this operator from the official helm chart because I really like the added flexibility however I am having the following issue.
When I try to create a cluster the bootstrap pod gets stuck with the error:
[2023-11-27T11:58:01,529][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-bootstrap-0] Node started
[2023-11-27T11:58:01,530][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-bootstrap-0] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2023-11-27T11:58:01,531][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-bootstrap-0] 0 OpenSearch Security modules loaded so far: []
[2023-11-27T11:58:01,535][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-bootstrap-0] Background init thread started. Install default config?: false
[2023-11-27T11:58:01,539][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-cluster-bootstrap-0] Wait for cluster to be available ...
[2023-11-27T11:58:01,555][INFO ][o.o.a.c.HashRing         ] [opensearch-cluster-bootstrap-0] Node added: [Tml4PMg-Syy2rfjh__oILQ]

[...]

[2023-11-27T11:07:22,030][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,033][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,033][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,034][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,034][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,035][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,035][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,036][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,036][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,036][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-cluster-bootstrap-0] Failure no such index [.opendistro_security] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-11-27T11:07:22,169][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[2023-11-27T11:07:22,180][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)
[2023-11-27T11:07:22,188][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-bootstrap-0] Not yet initialized (you may need to run securityadmin)

Because of this the first pod of the StatefulSet gets stuck (with the following log) and thus the other two replica are not created.
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.8.0
Detected OpenSearch Security Version: <IP_ADDRESS>
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin

I am hoping somebody here might have a clue as to what I am missing.
This is my config (everything is deployed in "monitoring" namespace including operator and secrets):
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: opensearch-cluster
spec:
  general:
    serviceName: opensearch-cluster
    version: 2.8.0
    setVMMaxMapCount: true
  security:
    tls:
      http:
        # TODO Swap to cert-manager generated ones due to expiration issue
        generate: true
      transport:
        generate: true
        perNode: true
  dashboards:
    enable: true
    version: 2.8.0
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
    tls:
      enable: true
      generate: true
  nodePools:
    - component: nodes
      replicas: 3
      diskSize: "5Gi"
      persistence:
        # The production system would use longhorn, but I have also tried emptyDir just in case and it has the same issue
        # emptyDir: {}
        pvc:
          storageClass: longhorn
          accessModes:
            - ReadWriteOnce
      resources:
         requests:
            memory: "512Mi"
            cpu: "100m"
         limits:
            memory: "512Mi"
            cpu: "100m"
      roles:
        - "cluster_manager"
        - "data"

I have also tried adding a securityConfig (in case the default internal_user.yaml is for some reason not deployed) as defined in this example with the corresponding adjustments to the admin password hash and my yaml
security:
    config:
      securityConfigSecret:
        name: securityconfig-secret
      adminCredentialsSecret:
        # opensearch-admin-secret was manually created with username and password = "admin"
        name: opensearch-admin-secret

Hi @f3l1x98
This looks like the securityconfig did not get initialized. Please check if the securityonconfig-update-job is running and has completed, also check its logs if it had a problem.
One potential problem that I see is that you give the nodes very little resources, both CPU and memory. This might make the first node pod too slow to come up before the health probes restart it. Please try with higher resources (I would suggest 1000m CPU and 2048Mi memory).

@swoehrl-mw I have managed to get the security config initialized.
Your guess was right, as soon as I increased the cpu resources it worked.
Thank you :+1:

Your guess was right, as soon as I increased the cpu resources it worked.

Good to hear, closing this issue.

@swoehrl-mw  i got same issue, and increassing cpu didn't solve it !


How can I disable visibility of voice search icon ?

because displayVoiceButton() is private, So not able to access this method.

You can just use setVoiceSearch(false) in order not to display voice search.


CFScript function length check is off by one

v1.5 (via commandbox-cflint, v2.3)
Here's a gist with a CFScript function that is 100 lines long.
I am using this .cflintrc file, in the same directory:
{
    "rule": [ ],
    "excludes": [ ],
    "includes": [
		{"code": "EXCESSIVE_FUNCTION_LENGTH"}
	],
    "inheritParent": false,
    "parameters": {}
}

Here's the output of my test run:
root@3200b7c82b5e:/app# wc -l test/cflint/script.cfm
102 test/cflint/script.cfm
root@3200b7c82b5e:/app# box cflint test/cflint/script.cfm

        Total Files:    1
        Total Lines:    103
        WARNINGS:       1

        /app/test/cflint/script.cfm   1
                WARNING: EXCESSIVE_FUNCTION_LENGTH, Function scriptOnlyFunction100Lines is 101 lines. Should be fewer than 100 lines. [2,1]
root@3200b7c82b5e:/app#

I can confirm this behavior.


404 at /en/latest/installation/windows/

I found a broken link : https://docs.docker.com/en/latest/installation/windows/

@wallyqs Here's the link on how to install Docker Desktop on Windows - https://docs.docker.com/docker-for-windows/install/.
Could you let me know how did you arrive at the link you've referenced?

Thanks for reporting; that looks to be a very old location that we used years ago
Given that these URLs are very old, we don't plan to add redirects for these, so I'll close this ticket, but feel free to continue the conversation


Lucy new feature

This looks good! Please add a unittest to confirm that nearby device count works when you have some data.

Done, thanks Danielle! Let me know if there is anything else I should add/change

First, it looks like some of the call tests don't pass. @carlan1 may be able to help with figuring out what is going on there. Second, please delete raw/wifi.py, unless I'm mistaken that feature doesn't exist anymore. Finally, please make sure to filter out wifi events from the count.

Fixed that, am editing docs now.


10 Connection with 5 minute testing time. Http server appear err: can not connect to server

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?
Any Version
What operating system and processor architecture are you using (go env)?
Window 10  64Bit
What did you do?
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
Testing tool : HttpTest4Net  with not keep-alive option
What did you expect to see?
What did you see instead?

There isn't enough detail to reproduce the issue.
Please answer the questions in the issue template (what did you do,
what did you expect to see and what did you see instead) in more
detail so that we can understand and reproduce the problem.
If you're just asking a question, please see
https://golang.org/wiki/Question.
Thanks.


The rarity does not work for me

Is the rarity actually working? Im not good at coding especially Js but if I look through the code I cant see where the superrare is used?

I am wondering the same. Is it just for metadata?

I am wondering the same. Is it just for metadata?

Right, the metadata shows up with the correct titles but it doesn't seem to actually make anything more rare. I'm trying to figure out how to code rarity into the program and I'll let you know when I figure it out.

@petercottonmouth it would be awesome if you can figure this out. Thanks!

The method I'm using is to add blank images to create the rarity within a directory. For instance, if I want a layer to be 1/100, I have 99 blanks along with the one image that contains something. This is just an artificial method for creating rarity (it works though), but the attribute will have a value (such as "blank35" when picking the 35th blank) in the metadata regardless of if the layer is blank or not.

I am wondering the same. Is it just for metadata?

Right, the metadata shows up with the correct titles but it doesn't seem to actually make anything more rare. I'm trying to figure out how to code rarity into the program and I'll let you know when I figure it out.

You just put _sr (super rare) or _r (rare) in the image filenames

You just put _sr (super rare) or _r (rare) in the image filenames

This alone does not make anything more rare. If you have one layer directory with only one file, even if you label it _sr, every single output image will contain it.

You just put _sr (super rare) or _r (rare) in the image filenames

This alone does not make anything more rare. If you have a layer directory with only one file, even if you label it _sr, every single output image will contain it.

Yes, it only affects the metadata property. To change the math you need to do something in the drawLayer function


refactor(v2): add missing theme-classic types

Motivation
Improve types for theme-classic
Have you read the Contributing Guidelines on pull requests?
yes
Test Plan
if compilation passes everything is fine
Related tickets
ref; #3424

generally i don't like type-aliases done in this way if we want to have typealiases for typescript we cold provide tsconfig
"paths": {
    "@theme/*": ["@docusaurus/theme-classic/lib-next*"]
}

instead of asking user include @docusaurus/module-type-aliases we could than ask him to use extends or specify paths by themself

Thanks :)
Note if you are interested to help us complete our TS types, we have this theme config that is currently very badly typed 😅
https://github.com/facebook/docusaurus/blob/master/packages/docusaurus-theme-common/src/utils/useThemeConfig.ts

@slorber i started working on configs already, but i think its better to do this in smaller batches as they are easier to review

yes definitively :) thanks


ctlv3/command: auto compact and defrag after "check perf"

$ ETCDCTL_API=3 ./bin/etcdctl endpoint status -w table
+----------------+------------------+-----------+---------+-----------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        |  VERSION  | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+-----------+---------+-----------+-----------+------------+--------------------+--------+
| <IP_ADDRESS>:2379 | 8e9e05c52164694d | 3.3.0+git |   20 kB |      true |         2 |          4 |                  4 |        |
+----------------+------------------+-----------+---------+-----------+-----------+------------+--------------------+--------+

$ ETCDCTL_API=3 ./bin/etcdctl check perf --auto-compact --auto-defrag
 60 / 60 Boooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s
Compacting with revision 8994
Compacted with revision 8994
Defragmenting "<IP_ADDRESS>:2379"
Defragmented "<IP_ADDRESS>:2379"
PASS: Throughput is 150 writes/s
PASS: Slowest request took 0.146769s
PASS: Stddev is 0.002704s
PASS

$ ETCDCTL_API=3 ./bin/etcdctl endpoint status -w table
+----------------+------------------+-----------+---------+-----------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        |  VERSION  | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+-----------+---------+-----------+-----------+------------+--------------------+--------+
| <IP_ADDRESS>:2379 | 8e9e05c52164694d | 3.3.0+git |   20 kB |      true |         2 |       8998 |               8998 |        |
+----------------+------------------+-----------+---------+-----------+-----------+------------+--------------------+--------+

/cc @raoofm @spzala
For https://github.com/coreos/etcd/issues/9326.

@raoofm I added --auto-compact flag, as an option. It will discards all entries before the revision from last delete operation. It's optional, in case check perf commands are run mixed with other data (it will discards previous keys with previous revisions).

@gyuho this is very useful and it looks great. Couple of comments - 1. can we have both of these as a func on its own in util.go or at least in check.go if you think it's a good idea for re-usability? I believe I should use these options with check datascale as a follow up PR on my PR under review too and can simply call these functions. 2. I have noticed in few place in doc, that compact and defrag typically follow up with etcdctl alarm disarm, we may not need it here but just a thought if it's good to add disalarm as an option too. Thanks!

Codecov Report

Merging #9330 into master will decrease coverage by 0.36%.
The diff coverage is 9.52%.

@@            Coverage Diff             @@
##           master    #9330      +/-   ##
==========================================
- Coverage   75.81%   75.44%   -0.37%
==========================================
  Files         365      365
  Lines       30695    30715      +20
==========================================
- Hits        23270    23173      -97
- Misses       5814     5921     +107
- Partials     1611     1621      +10

Impacted Files
Coverage Δ

etcdctl/ctlv3/command/check.go
13.44% <9.52%> (-0.7%)
:arrow_down:

clientv3/ordering/kv.go
16.17% <0%> (-69.12%)
:arrow_down:

pkg/transport/timeout_conn.go
80% <0%> (-20%)
:arrow_down:

pkg/adt/interval_tree.go
85.28% <0%> (-8.71%)
:arrow_down:

etcdserver/api/v3rpc/lease.go
78.37% <0%> (-8.11%)
:arrow_down:

pkg/fileutil/purge.go
73.68% <0%> (-7.9%)
:arrow_down:

clientv3/namespace/watch.go
87.87% <0%> (-6.07%)
:arrow_down:

clientv3/concurrency/session.go
88.63% <0%> (-4.55%)
:arrow_down:

rafthttp/peer.go
90.22% <0%> (-3.01%)
:arrow_down:

etcdserver/api/v3election/election.go
64.7% <0%> (-2.95%)
:arrow_down:

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e9ba31...87d1e99. Read the comment docs.

@spzala I separated as a function (let's keep them in check.go for now).
Can you create a follow-up PR to use this in other commands?
Thanks!

@gyuho yup, that's perfect. Thanks!!
lgtm

@gyuho is there a plan to backport to 3.1 and 3.2, if not can we?

@gyuho Is this expected?
# docker exec 4a5ce770e887 sh -c 'ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=<IP_ADDRESS>:2379 version'
etcdctl version: 3.3.9
API version: 3.3

# docker exec 4a5ce770e887 sh -c 'ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=<IP_ADDRESS>:2379 check perf --auto-compact --auto-defrag'
Error: unknown flag: --auto-compact

docker exec 4a5ce770e887 sh -c 'ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/peer.crt --key=/etc/kubernetes/pki/etcd/peer.key --endpoints=<IP_ADDRESS>:2379 check perf  --auto-defrag'
Error: unknown flag: --auto-defrag

@danbeaulieu I got the same error, can not use flag --auto-compact and --auto-defrag

@danbeaulieu @dreambo8563
Changes about --auto-compact is in etcdctl/ctlv3/command/check.go. You can view source code. The changes are not in branch release-3.x but master.


Prioritisation network manager + transactions manager + eth request handler

Closes https://github.com/paradigmxyz/reth/issues/6589. Closes https://github.com/paradigmxyz/reth/issues/6336.
Generalises preemption pattern from https://github.com/paradigmxyz/reth/pull/6334 (originally from NetworkManager when polling Swarm). Incorporates this pattern into NetworkManager as well as TransactionsManager.
Decomposed into https://github.com/paradigmxyz/reth/pull/6656 and https://github.com/paradigmxyz/reth/pull/6651 and this pr.

Benchmarks of PR for the past 2 days with maxperf profile. Restarts are visible roughly for each new commit from past 2 days. Since NetworkManager isn't spawned in a blocking thread in /bin/reth, and other crates implement the blocking async interface that this pr is replacing for network crate, NetworkManager now poses a higher risk of being starved by threads running other node components.

I'd like to limit this pr to transaction manager only first

the reason this pr became combined network manager + tx manager + eth request handler, is because when there are three tasks spawned as non-blocking, but are doing working in a blocking way, then you unblock one of those tasks? that one task which has been unblocked will starve. logically, since the other two are still hogging resources. I doubted whether this pr was large enough or if it would have to encompass all the other threads in the codebase that are spawned as non-blocking. however, since I benched marked it, it seems fine. ideally we would stretch this pr to cover all threads however. at best then we get it over the line quickly because anything that covers a big part of the codebase is subject to end up chasing main, i.e. a lot of work to fix merge conflicts correctly (all too often parts of important commits go missing on fixing merge conflicts).

I like the granular approach to budgeting nested streams, but imo the macro makes this unpleasant to read, I'd prefer doing this without the macro even if this means a bit more code

it doesn't need to be a macro if it's hard for you to read. I would make it a function instead that will receive closures to execute, since functions encompass code that would otherwise be duplicate, this improves readability since you can say with one glance at the method signature (or macro invocation) "aha, the same code is running everywhere". basically "the same pattern is employed in various places".

dug up some helpful resources
tasks can starve in tokio, just "not for too long":
https://github.com/tokio-rs/tokio/blob/5658d7c5032ade1bd0b18e3ec5b2b788eeac630e/tokio/src/runtime/coop.rs#L45-L57
pre-emption in a loop can cause busy-wait:
https://doc.rust-lang.org/nightly/std/thread/fn.yield_now.html
(Tokio's yield_now is analogous to that of std lib https://docs.rs/tokio/latest/tokio/task/index.html#yield_now)

merged into main at commit 3e54d9415e47d9393e738c30a8438dc1ec946046. looking a lot better than on main.

the results look great.

yay! are you keen on getting this into beta or no?
if so, it would be nice if time can be allocated to extended the prio-poll pattern to ActiveSession. each active session is spawned as a non-blocking task. this could possibly reduce spikes in FetchEvents.

I think I can live with the macro, if we rename it to poll_nested_stream_with_budget! because I find it yield point misleading because the macro doesn't trigger a yield point (return Pending)

true, good point. naming that is as coherent with the code as possible is key to readability and adoptability.

yay! are you keen on getting this into beta or no?

definitely


Disable NGINX OTEL tracing for specific path(s)

This tracing results in around 200-300 spans per second for posting metrics, which results in a lot of noise. It is useful to have tracing enabled for nginx when the applications are used, since services might communicate with each other through nginx, and then we want the trace information to be propagated over nginx. Ideally we would disable tracing for the metrics endpoint only.

@slarwise can't remember exactly what this is about.
Can you share a print screen or something.
Do I understand you correctly that each time the health endpoint of ingress-nginx is queries datadog generates a traces?

NGINX DataDog specific tracing is deprecated in favor of OTEL.


Remove unwanted files

These were added by #15472, I assume accidentally.

@BillWagner or @maira Can you review please?


Mafia III - GPU hang during loading

When I try to open Mafia III, it hangs during loading and I have to close the terminal to gain control over my system otherwise I will need to restart my computer. The game runs with Wined3d but everything is so dark and I can't see anything.
PS: I run the game in window mode not full-screen mode because it is a known reason for this game to not start with DXVK.
Software information
Mafia 3 - 1366× 768 - Low settings.
System information

GPU: AMD HD 8750M
Driver: Mesa-git 18.3
Wine version: TKG-3.19
DXVK version: 0.91

Log files

d3d11.log: Mafia3_d3d11.log
dxgi.log: Mafia3_dxgi.log
hang_report.txt

You have a very good memory. I upgraded the kernel from V. 4.14 to 4.19 and that fixed transform feedback issues with The Witcher III but that broke Mafia III.
I reported it here because of this open discussion >>> https://github.com/doitsujin/dxvk/issues/252
I already have an opened bugs at Mesa, and Wine, but they are moving slowly. I really regret having a laptop with an AMD card. It was a big mistake.

Hey, just received a SI card (AMD Radeon HD 7800 Series), I should be able to fix most of the issues soon.

Thanks.


feat: persist project config in deno.json

My understanding is that it reads a config file and merges it to Args. But I'm a little bit concerned with this approach because Args is just a parsed result of std's parse function, which doesn't have any domain knowledge of individual applications.
What I'd like to propose is to define a new class/object that stores all configurable values (which I'll call Config) and construct Config from the config file, command line args, and even env vars, then we'll be able to use the Config instance throughout the process.

I don't disagree, but I'd rather do it in a separate PR, as this work would be orthogonal to the scope of this PR. I've created the issue https://github.com/denoland/deployctl/issues/196 for it

Question: will we add support for JSONC later?

that's the plan, yes. I've created an issue also for this to keep track of it: https://github.com/denoland/deployctl/issues/198


Fix pull request number 362 in changelog

Very simple change, I copied the wrong pull request number into #362 by accident.

Coverage remained the same at 99.132% when pulling a1a1f01e9ba7011cc15616329d0f8796c5d8edcb on isaacbrodsky:fix-pr-no-362 into de1a7cb27891ed8b9934288ee645a4b307553c76 on uber:master.


Override linkage in manifest

I've been working with vcpkg in commercially distributed software. As such I need to watch the license of packages to determine how I can link them with our software. For example ffmpeg is LGPL so it needs to be linked as a shared library to not violate the L-GPL license.
Though I like to use as much as possible thru static libraries to allow compilers to optimize code in the entire program including 3rd party libraries and keep distribution a bit easier. Right now I'm using a custom vcpkg triplet or actuall an overlay of the default triplet names to 'fix' linkage based on port names.
This sort of works, but it requires a complete rebuild of all my dependencies everytime I'm adding a new entry to my manifest that doesn't conform to my default library linkage. This is annoying.
Proposed solution
It would be great if I could override for a port that it needs to be linked different.
Describe alternatives you've considered
Something else that could be interesting is to have information about the license in the CONTROL file of each port. With access to the license string then also in the triplet one could write something like:
if(${LICENSE} MATCHES "LGPL")
  set(VCPKG_LIBRARY_LINKAGE dynamic)
endif()

But I assume that the CONTROL files are not parsed before reading the triplet config - didn't check that yet.

Ping @strega-nil lest I forget something, we don't have this feature, right?

Configuration outside scripts, triplets and portfiles may create problems with binary caching. So I'm not sure if the following should be recommended: AFAIU only the triplet file content itself is part of the ABI tracking. What if you use an include statement to load specialized configuration, at least during development? You would still need to carefully track what is in the build at release time.
A versioned name of the included file, or a comment in the tripet, may help to rebuild the world when needed.
(But license tracking is an issue which perhaps needs more attention. With a GPL app, my desire is to keep out features which are not compatible with GPL. I think will need some override ports with a 'GPL ' feature'. 'LGPL' might be an alternative feature.)

It's been a while when I was working with it but I don't think you can include files from the triplet files. Because I wanted these changes for all our platforms I was trying to combine the checks into a single file and include that from the triplet files.
Didn't get that to work back then but things might have changed since.


ADF pipeline in toolkit v0.2 breaks due to syntax errors in set scope and set date step

🐛 Problem
ADF pipeline "msexports_ETL_ingestion" the steps "Set scope" & "Set date" outlining the following syntax error: Syntax error: Missing period. This breaks the pipeline execution
👣 Repro steps
open the pipeline, select the settings tab, the value is marked red
🤔 Expected
execution of the step in the pipeline succeeds
📷 Screenshots

🙋♀ Ask for the community
the line of code looks like the following:
@replace(split(pipeline().parameters.folderName,variables('folderArray')[sub(length(variables('folderArray')), if(greater(length(variables('folderArray')[sub(length(variables('folderArray')), 2)]), 12), 3, 4))])[0],'msexports','ingestion')
error: Position 196 Syntax error: Missing period
I assume that their should be a period between ('folderArry') and [sub(length.... however this even not work
We could use your help:

Please vote this issue up (👍) to prioritize it.
Leave comments to help us solidify the vision.

@DUeffing I saw the same error in the ADF but I wasn't able to pinpoint the actual issue and everything was working as expected. I assumed it was a bug in ADF.
@MSBrett do you see anything?

@flanakin thanks for the feedback, for the sack of clarity. I used the following source for the deployment:
https://github.com/microsoft/finops-toolkit/releases/tag/v0.2 and insight of it the [finops-hub-v0.2.zip] file.

Thanks for confirming. This code is new in 0.2, so that's why I didn't ask 🙂

thanks for the feedback and I used copilot for explaining me what the code is doing and as well how to fix it. The explanation was good, but the proposal to fix it not. Nevertheless, the variables must be set otherwise the destination path for the parquet file will not set correct which breaks the entire pipeline. Any ideas how to fix it? Otherwise I would switch back to the 0.1 version of the pipeline as this worked as well with FOCUS data.

I see the same error in v0.2.
In the set Date there is a similar error:
@substring(variables('folderArray')[sub(length(variables('folderArray')), if(greater(length(variables('folderArray')[sub(length(variables('folderArray')), 2)]), 12), 2, 3))], 0, 6)
I suggest use the set options I mention here instead of using the folderArray variable:
https://github.com/microsoft/finops-toolkit/discussions/523

@DUeffing wait, is it breaking for you? I see the error in the ADF portal, but the code runs fine. Are you saying it's failing for you?

Hi @flanakin , I tested it again and the pipeline succeeded. However, I recognized that the event grid subscriber does not exists anymore and therefore the pipeline does not starts automatically. I could only start the pipeline by providing the values for the folder and the file manually in ADF for the trigger.

I recognized the same issue with the syntax error. In my case the pipeline is running fine but the trigger has not been active after deployment:

Hi, the trigger is out of the box deactivated / stopped obviously. When activating the trigger, it requires to publish the change to the pipeline for getting effective. Which then in turn would create the corresponding event grid subscription (as an requirement for the trigger itself). However, this is not working as the publishing failed due to the syntax error described beforehand

@flanakin any thoughts on this one?

Sorry, wasn't sure if you were aware of the deactivated trigger. Imo in version 0.1 the trigger was active by default.

Any update on when the ADF pipelines will be fixed. I redeployed yesterday and still the same issue.

Here's a release candidate: https://gist.github.com/flanakin/419b5ed778d12a109b297d0dd406c10f
I did a quick pass to make sure the files land in ingestion. Will confirm the rest later.

thanks @flanakin, will test it accordingly and let you know

@DUeffing Made some updates to fix a path issue and validated using the Cost summary report. Please let me know if you run into any issues.
https://aka.ms/finops/hubs/rc


[FLINK-21996][coordination] - Part one: Tests and adjusted threading model

Purpose of this PR
This is the first part of the fix for FLINK-21996.
(1) Refactoring of Number Sequence Source
The initial commits refactor the Number Sequence Source to increase the flexibility.
The source was previously very inflexible any could only deal with one split per parallel reader.
To reduce future maintenance for the tests, I wanted to reuse this existing source and thus needed to increase the flexibility a bit (which is also good for that source in any case).
(2) Test for RPC message loss
The big commit 6360d65 introduces an Integration Test Case that simulates lost RPC messages and lost RPC acks for the events that assign source splits. As expected, this currently violates exactly-once semantics.
The main test is in OperatorEventSendingCheckpointITCase and uses some tricks to inject an RPC gateway decorator to the MiniCluster, which can then be used to set various filters for the RPC calls. Some test methods are annotated with @Ignore, because they would detect the current shortcoming and fail. The @Ignore will be removed once the follow-up fix is in.
(3) Operator Coordinator Event sending threading model
Previously, events sent from the OperatorCoordinator to the tasks went synchronously to the Job Vertex and the latest Execution and the TaskManager RPC Gateway. However, this leads to hard-to-handle situations with races between event sending and task failures and recoveries. In extreme cases, an the Coordinator's thread could trigger an event to be send, and before the event reached the Execution a failure and recovery happened, meaning the event went to a later Execution than intended.
Commit eea2bac changes the threading model so that event to be sent from an OperatorCoordinator to a task are now passed into the Scheduler Main Thread and sent from there. Because we need to preserve the order of events and checkpoints, the checkpoint completion is also passed into the main thread now.
This does not fully fix the above-mentioned possible race, but it simplifies parts to make the next fix easier.
Verifying this change

This change is covered by existing tests and adds a new test.
To verify the OperatorEventSendingCheckpointITCase, remove the @Ignore annotation and see it fail: The program never completes because it waits for data that got lost with the lost RPC calls.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? no
If yes, how is the feature documented? not applicable

@kezhuw Thanks for the review.
The overview of how FLINK-21996 will be fixed comes in the next PR - this one is purely setting up tests, and I made it a separate PR because the test is quite elaborate and needs some specific tricks (RPC intercepting).
I will go through your comments.

@kezhuw I addressed your comments, except for the suggestion with the builder, where I am not quite sure how to address this.

@StephanEwen HI, all my concerns are either solved or answered. From my side, it is in a merge-able state. It still be good and helpful if @tillrohrmann and/or @becketqin could give another eyes on.

Thanks for the thorough review, @tillrohrmann.
I am applying your suggestions now.
About the result futures for event sending: Having the TaskNotRunningException in the futures doesn't regress here, because that exception was never possible before in how we use the coordinators. A follow-up PR (#15605) ensures the failure cases are taken care of, so these futures are just information for users.
We may even change the method to return void in the future (no pun intended), I just kept the returned CompletableFuture for now because they might come in handy when we want to send events with relaxed guarantees (for events that don't need exactly-ones semantics).

@tillrohrmann I incorporated your changes, except where I think it doesn't work as mentioned in the comments.

Manually merged.


Cant get it working

Sorry to bother you - please delete if not wanted, but im really stuck with getting this to work. It used to work months ago but now its broken. Have a log below DSM>docker - can`t work out what to do
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Starting main loop. Accepting connections.
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Setting the various signals.
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Not running as root, so not changing UID/GID.
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: listening on fd [4]
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: trying to listen on host[::], family[10], socktype[1], proto[6]
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: listening on fd [3]
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: trying to listen on host[<IP_ADDRESS>], family[2], socktype[1], proto[6]
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: listen_sock called with addr = '(NULL)'
2024/06/27 20:40:18 | stdout | NOTICE    Jun 27 20:40:18.758 [1]: Reloading config file finished
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Disabling transmission of the "Via" header.
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Setting "Via" header to 'tinyproxy'
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Added basic auth user : xxxx
2024/06/27 20:40:18 | stdout | INFO      Jun 27 20:40:18.758 [1]: Stathost set to "tinyproxy.stats"
2024/06/27 20:40:18 | stdout | NOTICE    Jun 27 20:40:18.758 [1]: Reloading config file (/etc/tinyproxy/tinyproxy.conf)
2024/06/27 20:40:18 | stdout | NOTICE    Jun 27 20:40:18.757 [1]: Initializing tinyproxy ...
Happy to pay for some beers if anyone can help

That looks like a normal startup. Maybe something external changed like docker compose configuration or network setup?
Have you tried connecting to tinyproxy from the same host with e.g. curl -v http://localhost:8888 (replace port 8888 if needed) or curl -v http://<IP_ADDRESS>:8888 (or whatever your interface address is since connecting to localhost may not work in all situations, if I remember right).
Maybe also confirm that netstat -tlnp shows docker-proxy (if running with docker) listening in the expected address and port.
(Happy to receive some beer money, but not sure how :). I do have bitcoin and ethereum addresses listed in my profile for donations.)

Thanks Kalaksi, i`ll work out the bitcoin thing.
My results from running those commands:
tcp        0      0 <IP_ADDRESS>:8888            <IP_ADDRESS>:*               LISTEN
tcp        0      0 <IP_ADDRESS>:444             <IP_ADDRESS>:*               LISTEN
same on tcp6

GET / HTTP/1.1
Host: localhost:8888
User-Agent: curl/7.86.0
Accept: /

< HTTP/1.0 407 Proxy Authentication Required
< Server: tinyproxy/1.11.2
< Content-Type: text/html
< Proxy-Authenticate: Basic realm="Tinyproxy"
< Connection: close
<

407 Proxy Authentication Required

Proxy Authentication Required
This proxy requires authentication.

Generated by tinyproxy version 1.11.2.
With auth and username # out
< HTTP/1.0 400 Bad Request
< Server: tinyproxy/1.11.2
< Content-Type: text/html
<

400 Bad Request

Bad Request
You tried to connect to the machine the proxy is running on
Does that look normal?
extra lines in the log for the auth enabled one
2024/06/28 11:29:46 | stdout | CONNECT   Jun 28 11:29:46.946 [1]: Request (file descriptor 5): GET / HTTP/1.1
2024/06/28 11:29:46 | stdout | CONNECT   Jun 28 11:29:46.946 [1]: Connect (file descriptor 5): <IP_ADDRESS>

That confirms that tinyproxy is running and should work. The error about bad request is because of the HTTP request being sent directly to the proxy instead of using it as a proxy. Next, for a proper test you should try e.g. curl -v -x http://<IP_ADDRESS>:8888 https://duckduckgo.com
And don't worry about the bitcoin, it's probably not worth the hassle if you're not already familiar.

CONNECT duckduckgo.com:443 HTTP/1.1
Host: duckduckgo.com:443
User-Agent: curl/7.86.0
Proxy-Connection: Keep-Alive

< HTTP/1.1 200 Connection established
< Proxy-agent: tinyproxy/1.11.2
<

TLSv1.0 (OUT), TLS header, Certificate Status (22):
TLSv1.3 (OUT), TLS handshake, Client hello (1):

So, looks like its working locally. So is the issue going to be with my windows machine on the local network im trying to point through the proxy? I can access all the other service im hosting in this docker project (arrs, qbitorrent) over the local network.  I`m using SwitchyOemga to route traffic - are there further test I can do now from the client (windows PC)?
I`ll work the BTC thing out.

From the windows host, you could try to directly access the proxy's address using a web browser (similar to the first test we tried).
Since you're able to access other services on the same server, it's probably not a routing issue, so the issue could be a firewall. What is the problem exactly you're facing? A timeout or some kind of error?

I added a log above after your reply. Using chrome to access the local ip - port 444 cant be reached, unexpectedly closed. 8888 has been spinning for a while now (5 mins)
CONNECT   Jun 28 14:32:05.912 [1]: Established connection to host "<IP_ADDRESS>" using file descriptor 204.
-- | -- | --
2024/06/28 14:32:09 | stdout | INFO      Jun 28 14:32:05.911 [1]: opensock: getaddrinfo returned for <IP_ADDRESS>:8888
2024/06/28 14:32:09 | stdout | INFO      Jun 28 14:32:05.911 [1]: opensock: opening connection to <IP_ADDRESS>:8888
2024/06/28 14:32:09 | stdout | INFO      Jun 28 14:32:05.911 [1]: No upstream proxy for <IP_ADDRESS>
2024/06/28 14:32:09 | stdout | INFO      Jun 28 14:32:05.911 [1]: process_request: trans Host GET http://<IP_ADDRESS>:8888/ for 203
2024/06/28 14:32:09 | stdout | CONNECT   Jun 28 14:32:05.911 [1]: Request (file descriptor 203): GET / HTTP/1.1
2024/06/28 14:32:09 | stdout | CONNECT   Jun 28 14:32:05.911 [1]: Connect (file descriptor 203): <IP_ADDRESS>
2024/06/28 14:32:09 | stdout | WARNING   Jun 28 14:32:05.911 [1]: Maximum number of connections reached. Refusing new connections.

Make sure chrome doesn't do proxying when testing the connection to tinyproxy. Also, do you set MAX_CLIENTS yourself? I'm thinking maybe it's set too low.

Max Clients is at 100 amigo. Proxy auto detect is off and so is manual in windows. Chrome setting just open the windows ones.
Here is my yaml - the ports are opened in the Wireguard section - don`t mean to keep drip feeding info, but this version is running dsm in proxmox if it makes a diff.
ports: (wireguard section)
  - 8888:8888 # TinyProxy
  - 444:444 #tinyp https

tinyproxy:
image: docker.io/kalaksi/tinyproxy:latest
container_name: tinyproxy
# You can also use GitLab as an alternative to Docker Hub:
# image: registry.gitlab.com/kalaksi-containers/tinyproxy
restart: always
depends_on:
- wireguard
network_mode: "service:wireguard"
cap_drop:
  - NET_ADMIN
#ports:
  #- 8888:8888
  #- 444:444
  # See tinyproxy.conf for better explanation of these values.
environment:
  # Set to "yes" to disable the Via-header, set to empty to leave it enabled.
  DISABLE_VIA_HEADER: 'yes'
  # Set this to e.g. tinyproxy.stats to enable stats-page on that address
  STAT_HOST: tinyproxy.stats
  MAX_CLIENTS: 100
  # A space separated list:
  ALLOWED_NETWORKS: <IP_ADDRESS>/0 <IP_ADDRESS>/0
  # One of Critical, Error, Warning, Notice, Connect, Info
  LOG_LEVEL: Info
  # Maximum number of seconds idle connections are allowed to remain open
  TIMEOUT: 900
  # Username for BasicAuth
 # AUTH_USER: 'xxxx'
  # Password for BasicAuth (letters and digits only)
  # Prefer secrets-mechanisms instead of environment variables.
 # AUTH_PASSWORD: 'xxxx'

# secrets:
#   - auth_password
# # With Alpine base image, this is required, for some reason, or /run/secrets won't get mounted.
# tmpfs:
#   - /run
# healthcheck:
# test: ["CMD", "curl", "-I", "-H", "Host: tinyproxy.stats", "http://localhost:8888"]
#  interval: 5m
#  timeout: 10s
#  retries: 1
# volumes:
#   # If not using docker-compose for secrets, you can still manually mount a file containing the password.
#   - ./secrets/auth_password:/run/secrets/auth_password:ro

Secret for AUTH_PASSWORD
secrets:
auth_password:
file: ./secrets/auth_password

To me, it looked like the test connection to proxy was routed through the proxy itself too and causing a loop or something. But at least it looked like the connection to proxy arrived, so it looks like it should be working. What if you try to connect some other site through proxy? What does the tinyproxy log say?
Also, what is that port 444? Tinyproxy should only use 1 port, afaik.
(Btw, I too have some experience with wireguard and have noticed that wireguard might say that the tunnel is up and running when in fact something is wrong and nothing gets through. But in this case, I think we're already seeing the connection go through according to those logs.)

Hey man, hope you had a good weekend.
For some reason I though it needed a separate port for https :) - Wireguard VPN is up.
I can only get one site to work - which bizarrely is the only one that fails to load when not on a vpn (torrent site) and says - This site can’t provide a secure connection. Trying through the proxy works first time. Every other site times out.

2024/07/01 20:25:31
stdout
INFO      Jul 01 20:25:31.693 [1]: Not sending client headers to remote machine

2024/07/01 20:25:31
stdout
CONNECT   Jul 01 20:25:31.693 [1]: Established connection to host "www.workingsite.org" using file descriptor 10.

2024/07/01 20:25:31
stdout
INFO      Jul 01 20:25:31.670 [1]: opensock: getaddrinfo returned for www.workingsite.org:443

2024/07/01 20:25:31
stdout
INFO      Jul 01 20:25:31.645 [1]: opensock: opening connection to www.workingsite.org:443

2024/07/01 20:25:31
stdout
INFO      Jul 01 20:25:31.645 [1]: No upstream proxy for www.workingsite.org

2024/07/01 20:25:31
stdout
CONNECT   Jul 01 20:25:31.645 [1]: Request (file descriptor 9): CONNECT www.workingsite.org:443 HTTP/1.1

2024/07/01 20:25:31
stdout
CONNECT   Jul 01 20:25:31.645 [1]: Connect (file descriptor 9): <IP_ADDRESS>

And one that fails for good measure

24/07/01 20:26:55
stdout
INFO      Jul 01 20:26:54.544 [1]: Closed connection between local client (fd:23) and remote client (fd:24)

2024/07/01 20:26:55
stdout
INFO      Jul 01 20:26:54.262 [1]: Not sending client headers to remote machine

2024/07/01 20:26:55
stdout
CONNECT   Jul 01 20:26:54.262 [1]: Established connection to host "www.youtube.com" using file descriptor 18.

2024/07/01 20:26:55
stdout
INFO      Jul 01 20:26:54.240 [1]: opensock: getaddrinfo returned for www.youtube.com:443

2024/07/01 20:26:55
stdout
INFO      Jul 01 20:26:54.218 [1]: opensock: opening connection to www.youtube.com:443

2024/07/01 20:26:55
stdout
INFO      Jul 01 20:26:54.218 [1]: No upstream proxy for www.youtube.com

2024/07/01 20:26:55
stdout
CONNECT   Jul 01 20:26:54.218 [1]: Request (file descriptor 17): CONNECT www.youtube.com:443 HTTP/1.1

2024/07/01 20:26:55
stdout
CONNECT   Jul 01 20:26:54.218 [1]: Connect (file descriptor 17): <IP_ADDRESS>

Honestly at this point pal - i`m willing to ship you my PC if you thought you could fix it :)

Hmm, something working a little randomly and then timeouting sounds like an MTU size issue, but it's very hard to say. Do you set MTU manually in any wireguard configs (or for network interfaces or anything)?

Yeah, I try and set everything to 1500 where I can

1500 is normal for a regular network interface, but wireguard should have something like 1420 on it's interface (I think that's the default if not setting it in wireguard config), since the tunneling protocol has some extra overhead which gets added on top when packet leaves the regular interface.

No f'ing way, the MTU thing has fixed it!
Commented out the line in my Wireguard config where I'd forced it to 1500, so it now uses whatever default is and everything seems to be connecting. You ruddy legend! Thank you so much.


Move lambdas before STL section

This PR also improves the lambda slides in a few places and moves some slides from the lambda section back into the STL section.
Fixes: #246

[x] Merge #249 first

Looks good but for some reason I cannot load the artifacts to check the final result. Anyway, it also needs a rebase, so I'll retry after it

I rebased now.

Reading the slides through, things are not that simple... we use the STL quite a lot in the lamda examples actually !

very first example says :

std::vector<int> data{1,2,3,4,5};
std::for_each(begin(data), end(data), [](int i) {
std::cout << "The square of " << i
<< " is " << i*i << std::endl;
});

then we even use std::transform, non member begin/end and std::variant and std::visit in the last slide.
I fear it has to stay after the STL and even the "more STL"

The lambda chapter would definitely need more work to strip it of some of the STL content. @hageboeck what do you suggest?

I prefer having an easy introduction to lambdas, and putting them to good use in the STL chapter later. This way, the syntax is repeated often enough, and people learn how to use lambdas as in e.g.
STLFunc(iterator, iterator, [](Arg arg) {
  doSomething;
});

@hageboeck I tried to strip the STL part from the lambda chapter, added a few more examples at the end of the STL chapter and moved a few slides.


Improve Rust Hello World sample and tutorial

Received customer request to improve our Rust Hello World sample selection.

Another vote for a tokio-postgres example - the old non-async package has dependencies with vulnerabilities.
You're probably going to end up having some "fun" regarding async closures and variable capture, I arrived at this implementation:
async fn retry<T, F>(pool: &Deadpool, mut f: F) -> Result<T>
where
    for<'a> F: FnMut(&'a mut Tx) -> Pin<Box<dyn Future<Output = Result<T>> + 'a>>,
{
    let mut client = pool.get().await?;
    let mut tx = client.transaction().await?;
    let v = loop {
        let mut sp = tx.savepoint("cockroach_restart").await?;

        match f(&mut sp)
            .await
            .map(|t| async { sp.commit().await.map_err(Error::from).map(|_| t) })
        {
            Err(e) => match e {
                Error::PostgresTokio(e)
                    if e.code() == Some(&SqlState::T_R_SERIALIZATION_FAILURE) => {}
                _ => return Err(e),
            },
            r => break r,
        }
    }?
    .await?;

    tx.commit().await?;

    Ok(v)
}

Which can be used like so:
retry(&pool, |tx| {
    Box::pin(async move {
        // use tx
    })
})
.await?;

This is where the fun starts with moving values.


[uss_qualifier] Wire back in DSS interoperability v19 test

Existing DSS interoperability test is properly set up as common, and its v19 implementation created
In local_test config / CI, now both v19 and v22a tests run, and v19 contains the DSS interoperability test.

FYI putting this PR in draft until #119 is merged.


add tests in distribute plugin

Tests if '-mixed' is correctly trimmed or not trimmed from hbs file names when distributed.
All distribute tests pass.
This PR looks to include my intent to publish the next prepatch version.
I did not receive email so I'll have to do a manual publish after this PR merges.

Great work! I think you can revert all the package.json changes that are bumping the version and once this change is committed, run publish again.

Ok I rebased out that commit.  I think it's good now. Ty


Query on D3DDI and driver models

I was looking for information on D3DDI that DDrawCompat uses to fix many (which type?) of bugs.
Unfortunately my search only lead to 1 useful result at:
https://learn.microsoft.com/en-us/windows-hardware/drivers/display/direct3d
The information is inadequate that raised a few doubts:

I originally assumed DDC didn't work on XP because D3DDI was some DXGI/DX10+ feature. However the document mentions that D3DDI is supported since Win2K. Where did I get it wrong? Is it that XPDM and WDDM DDIs are different, and DDC only touches WDDM DDIs?

DDC doesn't work on VMs. What's the limitation there, is it interlinked to my previous point?

I saw DDC uses only ddraw.dll. Why is that? Is it because all the problems are all related to DDraw implementation and DDI, both of which are only linked to ddraw.dll?

3.a) If the above is true, then dgV2 has a d3dim.dll as well, is that file responsible for something completely different that DDC doesn't do (example could be the D3DRM demos that run fine with dgV2 files but DDC needs an old d3dim.dll file).

What is a good source to study on D3DDI? The Windows DDK? Both the new and older ones?

1.a)  If the last point in 1 was true, will older DDKs be useful for WDDM based DDIs on newer OS? Similarly does the new one contain stuff for XPDM based DDI?
Thank you.

I originally assumed DDC didn't work on XP because D3DDI was some DXGI/DX10+ feature. However the document mentions that D3DDI is supported since Win2K. Where did I get it wrong? Is it that XPDM and WDDM DDIs are different, and DDC only touches WDDM DDIs?

Yes, XPDM and WDDM are very different. WDDM is the new driver model developed for Vista. XPDM was still supported up to Windows 7, mainly for compatibility with legacy hardware that never received any WDDM drivers. Windows 8+ only supports WDDM. I haven't looked at the old driver model much and I have no plans to add support for it.

DDC doesn't work on VMs. What's the limitation there, is it interlinked to my previous point?

It might work, if WDDM is properly supported by the VM, but most WDDM drivers in VMs are still quite buggy, if they're available at all. I don't support VMs mainly for that reason. I had better luck with VMware than VirtualBox drivers when I was last experimenting with this, but support isn't perfect there either.

I saw DDC uses only ddraw.dll. Why is that? Is it because all the problems are all related to DDraw implementation and DDI, both of which are only linked to ddraw.dll?

ddraw.dll dynamically loads d3dim.dll or d3dim700.dll when Direct3D objects are created through DirectDraw. There is no need to replace the files, since the needed hooks can be installed from the ddraw.dll hooks. They mostly just implement a bunch of additional COM interfaces, so vtable hooks are enough.

If the above is true, then dgV2 has a d3dim.dll as well, is that file responsible for something completely different that DDC doesn't do (example could be the D3DRM demos that run fine with dgV2 files but DDC needs an old d3dim.dll file).

There is no d3dim.dll in dgVoodoo, you're probably referring to d3dimm.dll. I have no idea what's in it, you'd have to ask the author. Apps surely won't load it directly (as they don't with d3dim.dll either), it must be loaded and used by the ddraw.dll wrapper of dgVoodoo itself.
I don't know what demos you're referring to. If you need old d3dim.dll files for them, it's probably because of the broken Ramp device implementation in the Win 10+ d3dim.dll?

What is a good source to study on D3DDI? The Windows DDK? Both the new and older ones?

I usually just browse the references for the user-mode display driver functions here:
https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/d3dumddi/ns-d3dumddi-_d3dddi_devicefuncs
There is of course no tutorial or anything like that that I'm aware of. The reference is also quite spotty in places. I sometimes wonder if there is some additional private documentation or guides that driver vendors receive, because the information I could find in the public docs doesn't seem enough to implement a solid driver.
You can take a look at driver call traces of some apps to study how it fits together, which is also what I did. There is no logger for driver calls that I'm aware of other than DDrawCompat's debug logs though.

If the last point in 1 was true, will older DDKs be useful for WDDM based DDIs on newer OS? Similarly does the new one contain stuff for XPDM based DDI?

I don't recall reading any DDK docs specifically, either for XPDM or WDDM, so I've no idea what you'll find in different DDKs. I thought most information nowadays is found online, like the XPDM and WDDM related links already posted here. Is there something else in older downloadable DDKs which is not available in the online docs? I imagine the older DDKs would only contain information related to XPDM anyway.

ddraw.dll dynamically loads d3dim.dll or d3dim700.dll when Direct3D objects are created through DirectDraw. There is no need to replace the files, since the needed hooks can be installed from the ddraw.dll hooks. They mostly just implement a bunch of additional COM interfaces, so vtable hooks are enough.

I don't know what demos you're referring to. If you need old d3dim.dll files for them, it's probably because of the broken Ramp device implementation in the Win 10+ d3dim.dll?

These 2 questions stemmed from the Ramp problem. I originally assumed that, since the new Windows d3dim.dll is "stripped" in some sort, dgV2 has restored the function in d3dimM.dll (I will ask him what it does).
And that it was impossible for DDC to do it because it doesn't have a Direct3d separate module.
Now that you say a COM hook is enough, would that be enough to fix the Ramp issue.
(Not that I need a fix in ddraw now, just the way these things work).
Also assuming that this problem isn't even related to the driver, the fix wouldn't be necessary in DDI, right?

I don't recall reading any DDK docs specifically, either for XPDM or WDDM, so I've no idea what you'll find in different DDKs. I thought most information nowadays is found online, like the XPDM and WDDM related links already posted here. Is there something else in older downloadable DDKs which is not available in the online docs? I imagine the older DDKs would only contain information related to XPDM anyway.

I went and tried the DX7 DDK and found a few permedia files and files for some .c and headers for the reference driver. No useful documentation either.

These 2 questions stemmed from the Ramp problem. I originally assumed that, since the new Windows d3dim.dll is "stripped" in some sort, dgV2 has restored the function in d3dimM.dll (I will ask him what it does).

dgVoodoo doesn't use or need any of the ddraw-related system dlls. I also don't think it has an actual Ramp or even RGB software device implementation, most likely it only implements the HAL devices and uses them in place of the software devices too. So in essence, it's probably similar to what DDrawCompat does with the SoftwareDevice=hal setting.
Of course, I don't know its actual implementation (since it's closed source), I just assume this based on the fact that people use it to force things like resolution scaling and antialiasing on The Sims, which uses the RGB software device natively. I don't think this would be possible (at least with acceptable performance) on a pure CPU-rendered implementation.

Now that you say a COM hook is enough, would that be enough to fix the Ramp issue.

Not that simple. The whole Ramp SW renderer would have to be reimplemented from scratch most likely, and I don't think it's worth the effort, not to mention the exact details of its working are not really documented and it might be difficult to replicate precisely. The RGB device can emulate it pretty well anyway, except maybe for the monochromatic lighting. Well, maybe that too with some color manipulations, but I could never entirely figure out how the monochromatic lighting should differ from color lightning exactly, and which colors would need to be changed and how.
In theory, the Ramp device can also generate palette entries as long as the entire palette is not already filled, but I'm not sure if any games actually need that, since most of them seem to supply a complete palette already anyway.

Also assuming that this problem isn't even related to the driver, the fix wouldn't be necessary in DDI, right?

The Ramp device is a pure software rendering device. It doesn't even need a GPU. It is (or was) implemented in the DirectX runtime itself, without any driver involvement.

So in essence, it's probably similar to what DDrawCompat does with the SoftwareDevice=hal setting.

Good assumption, I think that's correct looking at the way it performs.
A last piece to the puzzle:
Wine has a ddraw directory in the source, but the d3dim and d3dim700 directories are pretty empty. Ddraw directory had connections to d3dlight, d3dviewport and such; but otherwise how the Wine d3d emulation works on Linux?

The Ramp device is a pure software rendering device. It doesn't even need a GPU. It is (or was) implemented in the DirectX runtime itself, without any driver involvement.

Does that mean it may be possible to get the ddraw files from an old Windows (XP or 7) installation and then shoehorn them (copy the dll's into the games directly and force the game to use these rather than native Windows' ones) somehow into an application that uses the Ramp device on a modern installation of Windows?  This way the game can work like it did in the past.

Wine has a ddraw directory in the source, but the d3dim and d3dim700 directories are pretty empty. Ddraw directory had connections to d3dlight, d3dviewport and such; but otherwise how the Wine d3d emulation works on Linux?

I'm not really familiar with Wine's implementation, but from a quick look they probably just implemented all the D3D functionality within ddraw.dll. Like I said, no games should actually require a separate d3dim.dll/d3dim700.dll, those are just helper DLLs for ddraw.dll that MS put in separate modules for some reason. It's an implementation detail not required to be followed by any alternative implementation.

Does that mean it may be possible to get the ddraw files from an old Windows (XP or 7) installation and then shoehorn them (copy the dll's into the games directly and force the game to use these rather than native Windows' ones) somehow into an application that uses the Ramp device on a modern installation of Windows? This way the game can work like it did in the past.

Yes, I tested this in the past and it seemed to work. It doesn't have to be that old, see here: https://github.com/narzoul/DDrawCompat/issues/22#issuecomment-616197074
From some more recent testing, I found that some of the very old D3D DLLs actually have worse performance on new systems even for HAL, so it's probably better to use the most recent working version.

Yes, I tested this in the past and it seemed to work. It doesn't have to be that old, see here: https://github.com/narzoul/DDrawCompat/issues/22#issuecomment-616197074

Just to clarify, you shouldn't use old versions of ddraw.dll itself, since that one may have stricter ties to newer WDDM versions and such, so old versions might not work at all. You should only use d3dim.dll, which is where the Ramp device implementation is.

Just to clarify, you shouldn't use old versions of ddraw.dll itself, since that one may have stricter ties to newer WDDM versions and such, so old versions might not work at all. You should only use d3dim.dll, which is where the Ramp device implementation is.

I see.  Thank you.

I'm not really familiar with Wine's implementation, but from a quick look they probably just implemented all the D3D functionality within ddraw.dll. Like I said, no games should actually require a separate d3dim.dll/d3dim700.dll, those are just helper DLLs for ddraw.dll that MS put in separate modules for some reason. It's an implementation detail not required to be followed by any alternative implementation.

Ok I compared stolen MS stuff vs Wine and that seems to be the case. That's all I needed
Also the DDKs (5.0 and 7.0) have a decent documentation for the DX working and the DDI. It is just that Windows 7 decided to install all files except the doc ones 🤦


TSPSolver.solve's time_bound arg has no effect

Hi,
I'm trying to use this for Kaggle's competition (197k cities). Is it possible that time_bound above a certain value has no effect? It works up to about 60.0. Above that, the solver never stops and I can't get the results from the solver.
Thanks a lot,
Yair

I have taken a look at the source code and there is indeed some code for checking the time bound but the algorithm is complex and there may be a loop somewhere taking a lot of time without checking it

I've noticed this too, and as @alsauve reports this is an intricacy with concorde itself. I will close this issue as wontfix since I am not sure how to address it, but please let me know if you have any ideas.

I've noticed this too, and as @alsauve reports this is an intricacy with concorde itself. I will close this issue as wontfix since I am not sure how to address it, but please let me know if you have any ideas.


Error when generating controller with composite Primary Key

There was an error when generating a controller for an entity with composite foreign keys, when generating, this error is printed in the console when executing the generate:
		##################################### Error #####################################

Primary key "id"  > 	Primary key "id" missing in Entity "null" (auto generated required)

Task Successful (total time: 28 seconds)

In the entity (the one with the composite key) controller, this is what's getting generated (get/id):
/**
    * GET /:Expression pkName is undefined on line 106, column 16 in unknown. :
    * get the "Expression pkName is undefined on line 106, column 37 in
    * unknown." entity_3.
    *
    * @param Expression pkName is undefined on line 108, column 17 in unknown.
    * the Expression pkName is undefined on line 108, column 31 in unknown. of
    * the entity_3 to retrieve
    * @return the Response with status 200 (OK) and with body the entity_3, or
    * with status 404 (Not Found)
    */
   @Path("/{Expression pkName is undefined on line 116, column 16 in unknown.}")
   @GET
   public Response getEntity_3(@PathParam("Expression pkName is undefined on line 118, column 53 in unknown.") Expression pkType
       is undefined on line 118, column 65 in unknown. Expression pkName is undefined on line 118, column 75 in unknown.) {
       log.debug("REST request to get Entity_3 : {}", Expression pkName is undefined on line 119, column 64 in unknown
       .);
       Entity_3 entity_3 = entity_3Facade.find(Expression pkName is undefined on line 120, column 66 in unknown
       .);
       return Optional.ofNullable(entity_3)
               .map(result -> Response.status(Response.Status.OK).entity(entity_3).build())
               .orElse(Response.status(Response.Status.NOT_FOUND).build());
   }

Please attach the modeler file.

Sorry forgot to attach.
TestPK.zip

Sure, it will be fixed soon.
Whatever you are expecting/missing from JPA, Session Bean, REST Controller will be done but in front-end we are using java hispter Angular JS template so UI side have some restriction like simple auto generated primary key named "id".
We will try to cover these types of limitation in Angular JS 2 version.
Fix for this jax-rs issue will be available ASAP.

Ok thank you, I'll be waiting.

Hi @evilkid ,
Generator for composite primary key (IdClass/Embeddable) is available for both REST and Arquillian.
** Generator for Derived Identities is still not developed.
Ref : http://jpamodeler.github.io/page.html?l=tutorial/DerivedIdentities

Awesome job, thank you.