config merge via imports overwrites the wrong part of the config

Hi,
I'm trying to use imports to append a custom config to the containerd configuration as i want to avoid editing the config.toml files
config.toml
# Kubernetes doesn't use containerd restart manager.
disabled_plugins = ["restart"]
oom_score = -999
imports = ["/etc/containerd/config.d/*.toml"]

[debug]
  level = "debug"

[grpc]
  gid = 412

[plugins.cri]
  stream_server_address = "<IP_ADDRESS>"
  max_container_log_line_size = 262144
[plugins.cri.cni]
  bin_dir = "/home/kubernetes/bin"
  conf_dir = "/etc/cni/net.d"
  conf_template = ""
[plugins.cri.registry.mirrors."docker.io"]
  endpoint = ["https://mirror.gcr.io","https://registry-1.docker.io"]

I have added a custom_config.toml file under /etc/containerd/config.d. It's content is

[plugins.cri.registry.mirrors."myregistrydomain.com"]
  endpoint = ["https://myregistrydomain.com"]
[plugins.cri.registry.configs."myregistrydomain.com".tls]
  ca_file = "/etc/custom-certs/cert.pem"

Before restarting the containerd daemon, a crictl info returns
{
  "status": {
    "conditions": [
      {
        "type": "RuntimeReady",
        "status": true,
        "reason": "",
        "message": ""
      },
      {
        "type": "NetworkReady",
        "status": true,
        "reason": "",
        "message": ""
      }
    ]
  },
  "cniconfig": {
    "PluginDirs": [
      "/home/kubernetes/bin"
    ],
    "PluginConfDir": "/etc/cni/net.d",
    "PluginMaxConfNum": 1,
    "Prefix": "eth",
    "Networks": [
      {
        "Config": {
          "Name": "cni-loopback",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "loopback",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"loopback\"}"
            }
          ],
          "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"
        },
        "IFName": "lo"
      },
      {
        "Config": {
          "Name": "gke-pod-network",
          "CNIVersion": "0.3.1",
          "Plugins": [
            {
              "Network": {
                "type": "gke",
                "ipam": {
                  "type": "host-local"
                },
                "dns": {}
              },
              "Source": "{\"ipam\":{\"ranges\":[[{\"subnet\":\"<IP_ADDRESS>/24\"}]],\"routes\":[{\"dst\":\"<IP_ADDRESS>/0\"}],\"type\":\"host-local\"},\"mtu\":1460,\"type\":\"gke\"}"
            },
            {
              "Network": {
                "type": "portmap",
                "capabilities": {
                  "portMappings": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"portMappings\":true},\"type\":\"portmap\"}"
            },
            {
              "Network": {
                "type": "bandwidth",
                "capabilities": {
                  "bandwidth": true
                },
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"capabilities\":{\"bandwidth\":true},\"type\":\"bandwidth\"}"
            },
            {
              "Network": {
                "type": "cilium-cni",
                "ipam": {},
                "dns": {}
              },
              "Source": "{\"type\":\"cilium-cni\"}"
            }
          ],
          "Source": "{ \"name\": \"gke-pod-network\", \"cniVersion\": \"0.3.1\", \"plugins\": [ { \"type\": \"gke\", \"mtu\": 1460, \"ipam\": { \"type\": \"host-local\", \"ranges\": [ [{\"subnet\": \"<IP_ADDRESS>/24\"}] ], \"routes\": [ {\"dst\": \"<IP_ADDRESS>/0\"} ] } }, { \"type\": \"portmap\", \"capabilities\": { \"portMappings\": true } },{\"type\": \"bandwidth\",\"capabilities\": {\"bandwidth\": true}},{\"type\": \"cilium-cni\"} ] }\n"
        },
        "IFName": "eth0"
      }
    ]
  },
  "config": {
    "containerd": {
      "snapshotter": "overlayfs",
      "defaultRuntimeName": "runc",
      "defaultRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "untrustedWorkloadRuntime": {
        "runtimeType": "",
        "runtimeEngine": "",
        "PodAnnotations": null,
        "ContainerAnnotations": null,
        "runtimeRoot": "",
        "options": null,
        "privileged_without_host_devices": false,
        "baseRuntimeSpec": ""
      },
      "runtimes": {
        "runc": {
          "runtimeType": "io.containerd.runc.v2",
          "runtimeEngine": "",
          "PodAnnotations": null,
          "ContainerAnnotations": null,
          "runtimeRoot": "",
          "options": {},
          "privileged_without_host_devices": false,
          "baseRuntimeSpec": ""
        }
      },
      "noPivot": false,
      "disableSnapshotAnnotations": true,
      "discardUnpackedLayers": false
    },
    "cni": {
      "binDir": "/home/kubernetes/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },
    "registry": {
      "mirrors": {
        "docker.io": {
          "endpoint": [
            "https://mirror.gcr.io",
            "https://registry-1.docker.io"
          ]
        }
      },
      "configs": null,
      "auths": null,
      "headers": null
    },
    "imageDecryption": {
      "keyModel": ""
    },
    "disableTCPService": true,
    "streamServerAddress": "<IP_ADDRESS>",
    "streamServerPort": "0",
    "streamIdleTimeout": "4h0m0s",
    "enableSelinux": false,
    "selinuxCategoryRange": 1024,
    "sandboxImage": "k8s.gcr.io/pause:3.2",
    "statsCollectPeriod": 10,
    "systemdCgroup": false,
    "enableTLSStreaming": false,
    "x509KeyPairStreaming": {
      "tlsCertFile": "",
      "tlsKeyFile": ""
    },
    "maxContainerLogSize": 262144,
    "disableCgroup": false,
    "disableApparmor": false,
    "restrictOOMScoreAdj": false,
    "maxConcurrentDownloads": 3,
    "disableProcMount": false,
    "unsetSeccompProfile": "",
    "tolerateMissingHugetlbController": true,
    "disableHugetlbController": true,
    "ignoreImageDefinedVolumes": false,
    "containerdRootDir": "/var/lib/containerd",
    "containerdEndpoint": "/run/containerd/containerd.sock",
    "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",
    "stateDir": "/run/containerd/io.containerd.grpc.v1.cri"
  },
  "golang": "go1.13.5",
  "lastCNILoadStatus": "OK"
}

After restarting the containerd daemon, somehow the config merge changes the binDir. below is a truncated output of crictl info to only the part i don't understand why it's changing
    "cni": {
      "binDir": "/opt/cni/bin",
      "confDir": "/etc/cni/net.d",
      "maxConfNum": 1,
      "confTemplate": ""
    },

What i don't understand is why my binDir is changing and where this value(/opt/cni/bin) coming from
Any help is appreciated it, happy to provide dumps

crictl version
Version:  0.1.0
RuntimeName:  containerd
RuntimeVersion:  1.4.4
RuntimeApiVersion:  v1alpha2

This is an expected behavior. When containerd server merges configuration from files defined in imports, it actually merges the plugin's configuration by section, not by key-value.
https://github.com/containerd/containerd/blob/05f951a3781f4f2c1911b05e61c160e9c30eaa8e/services/server/config/config.go#L313-L316
In your case, the configuration for plugins.cri is completely overwritten by  custom_config.toml, in which only registry mirrors are configured, everything else becomes un-configured, so the default values are used for them eventually.
The solution is moving the settings for cri plugin to one single toml file.

Is there any way to pass these custom values as KV inputs, while restarting containerd ?

@t-indumathy I don't think so, unless we change above-mentioned merge strategy for plugins' configuration, which is doable but I guess the current strategy is intentional. We need maintainers' opinions here.

I stumbled on this after running into a similar situation.
I've modified the config file for nvidia runtime and then added a runtime important for containerd mirrors.
It would be great to have a different merge strategy for the CRI plugin keys.

I don't understand why there would be an import function if you can't merge the imported structures into the configuration. This seems like a logical expectation when importing directories, especially. I would argue that it's not actually working as intended, just that it's working as implemented making this a valid bug that could be fixed.

The merge code is from https://github.com/containerd/containerd/pull/3574. There is a conversation in that PR about getting the merge right, and it looks like this approach won out:

What about just doing a replace based on the sections? I makes it easier to reason about and the user can just copy paste the section they want to change
-- https://github.com/containerd/containerd/pull/3574#issuecomment-524473330

I'm running into this issue as well.
I wanted to use a separate file to configure the registry config path (touching plugins."io.containerd.grpc.v1.cri".registry) and another one to configure the NVIDIA container runtime (touching `plugins."io.containerd.grpc.v1.cri".containerd).
The documentation at https://github.com/containerd/containerd/blob/main/docs/man/containerd-config.toml.5.md suggests this will work:

Imported files will overwrite simple fields like int or string (if not empty) and will append array and map fields.

However, as explained in the earlier comments, this is not true for plugins.  Is this something that can be addressed in the near future?

Hi, I have the same problem, the configuration from the imports is not merged with the config.toml file.

With 2.0 on the way (after we wrap up 1.7), I think this would be something worth looking at.  I do think that changing the merge strategy is a breaking change, but I see a couple options for 2.0:

Make this change and document it.  Possibly write out a warning the first time containerd 2.0 starts with a config that is merged differently, or vend a tool that can be used to validate upgrade conditions ahead of time.
Move to config version 3 that changes the merge strategy so that this is opt-in.

With 2.0 on the way (after we wrap up 1.7), I think this would be something worth looking at. I do think that changing the merge strategy is a breaking change, but I see a couple options for 2.0:

Make this change and document it.  Possibly write out a warning the first time containerd 2.0 starts with a config that is merged differently, or vend a tool that can be used to validate upgrade conditions ahead of time.
Move to config version 3 that changes the merge strategy so that this is opt-in.

Is there a timeline when 2.0 will be released? If it's a long way away option 2 could be already done right now as it wouldn't be breaking

To work around this we developed a utility to do toml key merging in https://github.com/d2iq-labs/capi-runtime-extensions of all files in /etc/containerd/custom-conf.d/*.toml and write it back out to /etc/containerd/config.toml.
Here is an example of a script that will
        #!/bin/bash
        set -euo pipefail
        IFS=$'\n\t'

        declare -r TOML_MERGE_IMAGE="ghcr.io/mesosphere/toml-merge:v0.2.0"

        if ! ctr --namespace k8s.io images check "name==${TOML_MERGE_IMAGE}" | grep "${TOML_MERGE_IMAGE}" >/dev/null; then
          ctr --namespace k8s.io images pull "${TOML_MERGE_IMAGE}"
        fi

        cleanup() {
          ctr images unmount "${tmp_ctr_mount_dir}" || true
        }

        trap 'cleanup' EXIT

        readonly tmp_ctr_mount_dir="$(mktemp -d)"

        ctr --namespace k8s.io images mount "${TOML_MERGE_IMAGE}" "${tmp_ctr_mount_dir}"
        "${tmp_ctr_mount_dir}/usr/local/bin/toml-merge" -i --patch-file "/etc/containerd/custom-conf.d/*.toml" /etc/containerd/config.toml

Is there a timeline when this will be fixed / v2 will be released? Because we keep running into this problem
This time it's because we switched to cgroupv2 and have to set SystemdCgroup = true, but that line of course gets overridden...

Removing from v2.0 milestone as staled


kernel/aarch64: implement interruptsEnabled

Something about reading DAIF?
https://github.com/CascadeOS/CascadeOS/blob/9e3a0422b1cf387a75803365d181734d09d49bbc/kernel/arch/aarch64/aarch64.zig#L34-L37

As the panic handler uses this functionality panicking is not a useful temporary implementation.
But as the aarch64 backend is not very far along with can get away with always return false for now.
https://github.com/CascadeOS/CascadeOS/blob/d689c2022350793f4580801a1ba53f5a746e0240/kernel/arch/aarch64/aarch64.zig#L34-L37

Closed in<PHONE_NUMBER>b5f8a1904c188e819fb60411240ec9


Rename Oliver's Place to Speakeasy

We call it Speakeasy elsewhere

defaults.txt is no longer alphabetical

Oh no! I'll fix that later
On Thu, 17 Nov 2022, 22:48 neil, @.***> wrote:

defaults.txt is no longer alphabetical
—
Reply to this email directly, view it on GitHub
https://github.com/kolmafia/kolmafia/pull/1298#issuecomment-1319307979,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABL4LUGBC5L463KZQQHJXDWI2Y2DANCNFSM6AAAAAASDZHSTI
.
You are receiving this because you modified the open/close state.Message
ID: @.***>


RECAPTCHACONTACT Header in YAML

I followed your instructions to create a custom languages.yaml file in the user folder. However, in oder for the site to work and not generate an error I had to add the following line in the header of the file just like the original file in the plugin ...
"RECAPTCHACONTACT:"
So, your example given in the instructions is missing the header (see sample below) ...
RECAPTCHACONTACT:
FORM_LEGEND: "Contact me"                       // Form Legend
SUBJECT: "New contact from Grav site!"          // Subject for email.
RECIPIENT<EMAIL_ADDRESS>          // Email address.
FIELDS:                     // Default fields, you can translate the text.
  NAME:
    LABEL: "Name"
    PLACEHOLDER: "Add your name"

  EMAIL:
    LABEL: "Email"
    PLACEHOLDER: "Add your email"

  MESSAGE:
    LABEL: "Message"
    PLACEHOLDER: "Add your message"

  ANTISPAM:
    LABEL: "Antispam"
    PLACEHOLDER: "Please leave this field empty for Antispam"

  SUBMIT:
    LABEL: "Submit"

MESSAGES:                   // Default messages, you can translate the text.
  SUCCESS: "Thank You! Your message has been sent."
  ERROR: "Oops! There was a problem with your submission. Please complete the form and try again."
  FAIL: "Oops! Something went wrong and we couldn't send your message."

Yeah, the example in the instructions is missing the header and the language itself. The user's languages.yaml file should have the same structure as the one provided in the plugin (you usually just copy the contents of the file to your custom languages files under the language you want and make the custom modifications, in this and any plugin with language support). I will change the README.md so that the instructions are clearer and have the RECAPTCHACONTACT: in them.
Thanks for letting me know.

Fixed. The update in instructions will appear in the next plugin release.
Thanks!

All working now! THX!

Glad to hear it! :+1:


Disable unused SDL loggers

By default SDL has a few loggers enabled (see log4cxx.properties file).
Three of them save data to files:

SmartDeviceLinkCoreLogFile => SmartDeviceLinkCore.log
TransportManagerLogFile => TransportManager.log
ProtocolFordHandlingLogFile => ProtocolFordHandling.log

Nevertheless ATF gets SDL logs trough Telnet logger which is not utilize files at all.
The idea is to disable some of the SDL loggers which writes data to file system.
This increase performance on environments with parallel test execution.

Closed via #176


Add firebase config files for deployment

Specific for carto project.

Already done :bow:

This was a previous test I did, so deleting


Feat: Adding Mem0 Integration

This pull request introduces integration with Mem0 memory storage, supporting both self-hosted and cloud versions. The following components have been added:

Mem0 Chat Memory: Retrieves and stores chat messages using Mem0 memory.
Mem0 Chat Memory Update: Updates messages in Mem0 memory storage.
Mem0 Chat Memory Delete: Deletes messages from Mem0 memory storage.

@NadirJ @rodrigosnader  Should we move this component to integrations/ bundles?

To merge the PR, I am removing the Update and deleting Mem0Component. Please connect with cc: @carlosrcoelho and @rodrigosnader , for further info. We could add it in the future if required.


Return more information in .getAttributes()

@asafkorem says in https://github.com/wix/Detox/pull/3009:

Just to point out, it seems like there are more attributes that we can provide with this API (see https://reactnative.dev/docs/accessibility), like accessibilityHint, or accessibilityState..
Sure, out of this PR scope. I'm talking about a change that requires both Android and iOS implementations.

@asafkorem, could you elaborate on this, please?
Don't hesitate to change the initial issue description or title if necessary.

Taking a look at the iOS implementation, it seems that we can simply provide more UIView's attributes, specifically of accessibility information (apple docs) like the mentioned accessibilityHint, or accessibilityState ..
I didn't looked at the Android code but there's no reason that we won't have these attributes easily accessible.
This will allow us to be correlated with RN's accessibility properties (https://reactnative.dev/docs/accessibility).

@asafkorem @noomorph let's reiterate. This sounds like it should be reopened.

Also relevant: https://github.com/wix/Detox/issues/3203 (see my comment).

Also, accessibilityRole (and probably accessibilitySubrole) could definitely be also relevant while we're at it (as suggested in #3301).

It's still a useful feature, even in 2023, any plans of supporting this feature soon?

Yes, please!
I am running into the issue as noted in #3203


alpha feature: PromptCritic

a simple, api key free way to improve your prompts before you send them.
https://twitter.com/swyx/status/1692694910871257476

we have a lot to improve on the UX of this, but need to ship. so slapping an alpha label on it


Change type of variables related to time to float

Summary

It closes #280.

Before this change, seekTo and currentTime worked with integer second.
It supports FLOAT SECONDS in the seekTo and currentTime.
I checked that seekTo and currentTime works on both iOS and Android.

Test/Review Plan

Code Review
Please test it works on the SAMPLE PROJECT.

Comment

Sorry for unclean commits. Please merge this by SQUASH MERGE.

@ohtangza


Split window and horizontal movement.

Splitting the window with the same buffer and having a line that doesn't fit in the window (nowrap) moving between them causes some rendering issues on the other side.

A simple solution was to add virt_text_hide = true in vim.api.nvim_buf_set_extmark at least it doesn't overwrite the text of the other windows.
I tried vim.api.nvim__add_ns but I was having to write a lot of code, it worked in parts but still writing to the same bufnr.

Neovim version (nvim -v)
NVIM v0.10.1

It's a limitation of how virtual text in Neovim works. I don't know of a way to fix this.
virt_text_hide mitigates this problem a bit, but because all indent extmarks start on the first column, it also means that you lose all indentation once you horizontal scroll any buffer. That is worse in my opinion.

oh ok! I got it, thanks!


NSDate being truncated to the second

NSDates are being truncated to a whole second when being stored and then retrieved from the realm. Looking at the date setter/getter, the NSTimeInterval is being cast to a std::time_t. NSTimeInterval is a double with sub second precision. My C++ is rusty but I believe time_t is a integer value of whole seconds. Could the underlying storage of NSDate be changed to a double?

how to convert a string date from json to NSDate or NSTimeInterval i am getting this:
*Invalid value '2014-12-13T12:00:00+01:00' for property

I have to agree with @dginsburg that having NSDate in the list of supported property types sends a mixed message. Until a resolution is found, perhaps the docs could link to the limitation detail directly in the property types list to be more explicit?

Realm supports the following property types: BOOL, bool, int, NSInteger, long, long long, float, double, CGFloat, NSString, NSDate (truncated to the second), and NSData.

@pizthewiz thanks for the suggestion, @yoshyosh just made the change on the website: http://realm.io/docs/cocoa/0.91.0/#property-types

Is it correct that the Realm database is storing the seconds but your Realm Browser app does not display them? Is there a way to see the seconds in the Realm Browser?

Is it correct that the Realm database is storing the seconds but your Realm Browser app does not display them?

Yes, seconds are being persisted, but the browser doesn't display seconds.
To see seconds in the browser, you could build it from source and change this line to use NSDateFormatterMediumStyle.

What's the status on NSDate property type being lossless? @jpsim

@marcoscurvello it's something that's still being discussed, since it would require changes at the core database level as to how we store dates. JP's earlier suggestion as to how to workaround the issue still stands:

In the meantime, if you need sub-second accurate dates, please use NSTimeInterval or any other supported Realm type with suitable accuracy.

@segiddins Okay, thanks. That's what I am currently doing. :smirk:

A question about NSTimeInterval: when I declare a property of type NSTimeInterval I can't use the dynamic keyword because its type cannot be represented in Objc. Is that ok (meaning that we won't have notifications about this property) or is there a way to workaround this?

@WolfTrinity are you sure you're not declaring that property as a NSTimeInterval? (Optional<NSTimeInterval>)? Objective-C primitives can't be represented as dynamic and Optional due to limitations in the Swift runtime.
If you need this property to be optional, you can mark it as RealmOptional<NSTimeInterval>.

@jpsim Oh sorry, my fault: I've just substituted NSDate? with NSTimeInterval?, but I forgot that not all the optional types are supported in the standard swift way, but need to use RealmOptional.
Anyway, this still prevents me to use dynamic (I can't declare dynamic let myDate =  RealmOptional<NSTimeInterval>()) so I can't use KVO on it.

You can't specify a RealmOptional property as dynamic:

There are two exceptions to this: List and RealmOptional properties cannot be declared as dynamic because generic properties cannot be represented in the Objective-C runtime, which is used for dynamic dispatch of dynamic properties, and should always be declared with let.

Source: https://realm.io/docs/swift/latest/#property-attributes
But they're still KVO-compliant.

Has this been assigned to a milestone / sprint? Seems JSON parsing is blowing up due to this

This is being actively worked on. It's hard to see how it would break JSON parsing though. Feel free to file a new issue with more details if you'd like some help understanding whatever bad behavior you're seeing.

That's great to hear it. Can't wait. Replacing the model with yet more DB shims to store NSTimeInterval but expose NSDates in the code is making our models look crazier than they already are.
Yes it's a place you would least expect it. But unfort when the truncation happens NSDate serialization returns without the decimal and the JSON parsing fails since it expects "eee dd-MMM-yyyy GG HH:mm:ss.SSS ZZZ" but gets "d MMM yyyy HH:mm:ss ZZZ". Maybe we're just using a crappy library (SwiftDate) but the downstream results happen none the less.

That sounds like a bug that should be fixed in your JSON handling either way. It's completely possible for a date to legitimately have no fractional seconds.


Expiration date is ignored in Chrome

In the Chrome browser, the Expiration date is ignored and the cookie is set as a Session cookie instead.
It may be because MaxAge is missing from the configuration?

I have just checked by setting a cookie in both static and interactive rendering - does not seem like this is the case for either of them.
We are going to need additional information in order to be able to investigate this further. An example solution where this can be replicated would be nice.
Also, consider checking the sample app for an implementation reference.

Closing due to a lack of activity.


Update to Italian translation (articolo -> post)

Any chance you could merge the recent work and re-submit?

Please let me know what's the recent work. This localization covers 100% on transifex so perhaps I'm missing something :)

Perhaps you mean I'm pulling from an old repository. I thought that wasn't a problem as long as there's no conflict or warning on GIT. But perhaps there might be differences in tools like po2php so I'll totally re-create my clone from origin removing all customizations for my site, at the end I'm not mantaining them and they become useless as hubzilla becomes more flexible.


Fix GitHub actions and unit tests

Hi, I noticed the CI was failing, so I've attempted to fix that under commit f9ec13d.
Once I had the CI running again there were two unit tests failing (test_in_operator and simple_in_operator_for_string), which were both failing due to filtering by enums using the 'in' filter.
These look like they had been failing for a while, I couldn't pinpoint exactly when they started failing though, I was getting xUnit errors when trying to run the tests on older commits, probably something I haven't set up on my machine though.
The problem seemed to be when you filtered by an array of enums, e.g. birthmonth ^^ ["January", "March"], it recognised the type as an enum, but just tried to run the array string directly through Enum.Parse. I've updated the logic to detect if it is an array in the same way arrays of strings are detected in the FilterParser, and then individually parse each Enum entry. This looks like it fixes it, but still required the unit tests to be updated to reflect the new types. This was all done under commit 783b387.
Finally I just wanted to make sure I hadn't broken any functionality with any of this, so I've added some integration tests to cover some Enum scenarios under commit 662ff59. The tests compare enums, but also against a faked title just to avoid getting a whole load of other entries with matching enums, not sure if this is the best way to do it, happy to change it if there's a better approach.
These changes around Enums will probably cause conflicts with my other merge #28, so I'll have to update that if you accept this pull request.
Hope these changes all make sense!
Thanks.

thanks! i was going to check it out before i merged all in, but appreciate it :-)

yeah i just ran main locally and they all seem to be passing there fwiw! regardless, i'm sure it's just minor tweaks and i appreciate all the effort with this 🍻


Resolve UI outlines using the correct target's viewport size

Objective
resolve_outlines_system wasn't updated when multi-window support was added and it always uses the size of the primary window when resolving viewport coords, regardless of the layout's camera target.
Fixes #14945
Solution
It's awkward to get the viewport size of the target for an individual node without walking the tree or adding extra fields to Node, so I removed resolve_outlines_system and instead the outline values are updated in ui_layout_system.

Not sure why the CI is failing with all those system ordering amibiguities.

Not sure why the CI is failing with all those system ordering amibiguities.

The CI is difficult to parse, you need to look for systems you specifically touched.

Yeah, I'm not thrilled with the output, but if the net effect is a decrease in ambiguities just set the expected number to the new total.


[Task]: Docs - Stardust Updates

Task description
With the upcoming stardust changes, some articles in the documentation may require updates and versioning.
And we need to be able to launch products with their respective docs to improve the overall developer experience.
Requirements
The following pages may require updating. Please feel free to add or remove any items you see fit.

[ ] Overview

[ ] Rust

[ ] Getting started

[ ] How to setup?
[ ] Examples

[ ] Guides

[ ] Guide X

[ ] API Reference

[ ] Node.js

[ ] Getting started

[ ] How to setup?
[ ] Examples

[ ] Guides

[ ] Guide X (excahnge)

[ ] API Reference

[ ] Java

[ ] Getting started

[ ] How to setup?
[ ] Examples

[ ] Guides

[ ] Guide X

[ ] API Reference

[ ] Python

[ ] Getting started

[ ] How to setup?
[ ] Examples

[ ] Guides

[ ] Guide X

[ ] API Reference

[ ] Troubleshooting
[ ] Contribute

A link to the parent issue can be found here: https://github.com/iotaledger/stardust-project/issues/45

Closed for https://github.com/iotaledger/wallet.rs/issues/1257, https://github.com/iotaledger/wallet.rs/issues/1258, https://github.com/iotaledger/wallet.rs/issues/1259 and https://github.com/iotaledger/wallet.rs/issues/1260


Unable to install pyproj on mac pro m1

Code Sample, a copy-pastable example if possible
Not being able to install pyproj on an m1 mac using pip install pyproj
Here's the error
Building wheels for collected packages: pyproj
  Building wheel for pyproj (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyproj (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      PROJ_DIR is set, using existing PROJ installation..

      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'pyproj._geod' extension
      pyproj/_geod.c:704:10: fatal error: 'geodesic.h' file not found
      #include "geodesic.h"
               ^~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyproj
Failed to build pyproj
ERROR: Could not build wheels for pyproj, which is required to install pyproject.toml-based projects

Problem description
I was successfully able to install proj using brew but can't seem to get this to work
Installation method

pip

Conda environment information (if you installed with conda):

Environment (conda list):

$ conda list proj

Details about  conda and system ( conda info ):

$<EMAIL_ADDRESS>

Probably need to install from source: https://pyproj4.github.io/pyproj/stable/installation.html#installing-from-source

Probably need to install from source: https://pyproj4.github.io/pyproj/stable/installation.html#installing-from-source

This does not seem to be possible:

https://github.com/python-poetry/poetry/issues/365
https://github.com/python-poetry/poetry/issues/1316

Probably related to generating universal osx wheels on Python 10: https://github.com/pyproj4/pyproj-wheels/issues/46
Side note: Due to the issues referenced, I don't use poetry for any geospatial python projects. Instead, I use pipenv as it allows installing from source.

You're probably right. The install works with poetry, but it picks the wrong architecture for the .so library and fails at runtime:
from pyproj._network import (  # noqa: F401 pylint: disable=unused-import
ImportError: dlopen(~/projects/PeopleForBikes/measure/.venv/lib/python3.10/site-packages/pyproj/_network.cpython-310-darwin.so, 0x0002): tried: '~/projects/PeopleForBikes/measure/.venv/lib/python3.10/site-packages/pyproj/_network.cpython-310-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/local/lib/_network.cpython-310-darwin.so' (no such file), '/usr/lib/_network.cpython-310-darwin.so' (no such file)

The workaround for this issue is the following:
rm -fr .venv/lib/python3.10/site-packages/pyproj*
poetry run pip install pyproj --no-binary pyproj

I had the same problem and here is what worked for me:
rm $HOME/venv/bin/proj
rm $HOME/venv/bin/pyproj

cd your_project_path
curl -sSL https://install.python-poetry.org | python3 -
poetry init # Accept all defaults
poetry run pip install pyproj --no-binary pyproj
rm pyproject.toml

Installing with poetry and the --no-binary option, as some of the workarounds above suggest, doesn't work for me:
❯ poetry run pip install pyproj --no-binary pyproj
Collecting pyproj
  Using cached pyproj-3.3.0.tar.gz (217 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      proj executable not found. Please set the PROJ_DIR variable. For more information see: https://pyproj4.github.io/pyproj/stable/installation.html
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

Will try to build from source next.

pyproj==3.3.1 was just released and wheels shouldn't be downloaded for Mac M1 anymore.
This should work:
brew install proj
python -m pip install pyproj

If it doesn't please let us know.

See: #1064

The above installation methods are not working for me on an M1, any advice?
Here is the error message I get, whether attempting to install via pip install pyproj or if trying to build from source per pip install pyproj --no-binary pyproj. (I use brew for proj):
 2 warnings generated.
      ld: warning: dylib (/opt/homebrew/lib/libproj.dylib) was built for newer macOS version (12.0) than being linked (11.0)
      ld: warning: dylib (/opt/homebrew/lib/libunwind.dylib) was built for newer macOS version (12.0) than being linked (11.0)
      ld: warning: dylib (/opt/homebrew/lib/libunwind.dylib) was built for newer macOS version (12.0) than being linked (11.0)
      ld: warning: ignoring file /opt/homebrew/lib/libproj.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
      ld: in '/opt/homebrew/lib/libunwind.dylib', building for macOS-x86_64 but attempting to link with file built for macOS-arm64
      clang-13: error: linker command failed with exit code 1 (use -v to see invocation)

This ended up working for me:
export PROJ_DIR=/opt/homebrew/Cellar/proj/9.0.0_1
 pip install pyproj --no-binary pyproj

If you have time to try out these wheels, that would be very helpful: https://github.com/pyproj4/pyproj/issues/1064#issuecomment-1184574399
Please upvote the comment if you have a positive experience and comment in #1064 if you get an opportunity to try it out and run into any issues.


fix support for rhel8/centos8

I've addressed the build/lint errors in #425. I'm happy to rebase this one when #425 is merged.

#425 was a duplicate, #418 was merged instead. So please rebase this PR if you are still interested in contributing.

@bbaassssiiee I just rebased my changes to the upstream/mater.

TODO: centos-6 can be removed.

@felixb & @bbaassssiiee
/vars/RedHat.yml already includes consul_os_packages for centos8
https://github.com/ansible-community/ansible-consul/blob/819f31dbe9cf35d566c7788416b57c468d5a2528/vars/RedHat.yml#L4-L14
Wouldn't it be cleaner to either add rhel to that list rather than specifying a whole new vars file for RedHat 8 or splitting them all up to vars files (Fedora.yml, OracleLinux.yml, etc ?)


ci: avoid ommiting releases

using skip ci in commit messages prevents releases when merged to the main branch.
Reference: https://github.com/semantic-release/git#merging-between-semantic-release-branches

:tada: This PR is included in version 1.1.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


list-of-modified-files.txt is empty when running cmorMDfixer-safe-mode-wrapper.sh

When running cmorMDfixer-safe-mode-wrapper.sh the 1st list with files to be modifed remains empty.
https://github.com/EC-Earth/cmor-metadata-fixer/blob/c68b5abfac988aa094a93fd9bf0869ba3b421ba3/cmorMDfixer-safe-mode-wrapper.sh#L60-L61
There is a typo in this line, see #1. Apart from that the command produces files list-of-modified-files.txt and cmorMDfixer-messages-1.log and yet tells:
All files in the entire dataset are correct, so  ./cmorMDfixer-safe-mode-wrapper.sh  will not apply any changes.

And list-of-modified-files.txt is empty, no files.
When replacing  >& cmorMDfixer-messages-1.log by  >& ${olist_1_filename} in the line above the cmorMDfixer-safe-mode-wrapper works fine, but it doesn't produce a cmorMDfixer-messages-1.log logfile any longer. Is that the solution? I don't fully understand the workings of the script yet.

Pinging @treerink
Even after retracting #1 I still have the same problem when running cmorMDfixer-safe-mode-wrapper.sh, the file list-of-modified-files.txt is empty,

I will have a look

Actually I can not repeat this (if understanding well). The first two lists are not empty, the last one (now the third one after the latest commit) should be empty, when talking about the cmorMDfixer-safe-mode-wrapper.sh script.
Could you try once more please and see if everything is alright concerning this issue?

Great, now everything seems to work, thanks a lot.
However, all list-of-modified-files-*.log files are still empty! The cmorMDfixer-messages-1.log contains the list of files to be processed, and similar cmorMDfixer-messages-2.log while cmorMDfixer-messages-3.log now just contains log messages (probably good).
What's the role of the list-of-modifed-files ? Do we need them?

I am a bit confused from what you report, therefore I list my testing:
First cleaning to be on the same page:
rm -f *.log list-of-modified-files-*; ./test-cmorMDfixer.sh clean

Run the default example on the test set with the safe-mode-wrapper:
./cmorMDfixer-safe-mode-wrapper.sh 1 metadata-correction-cases/knmi-metadata-corrections-piControl.json cmorMDfixer-test-data/test-set-01/CMIP6

which gives me the following content of files:
more list-of-modified-files-* *.log

::::::::::::::
list-of-modified-files-1.txt
::::::::::::::
cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr/v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v20240129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
::::::::::::::
list-of-modified-files-2.txt
::::::::::::::
cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr/v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v20240129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
::::::::::::::
list-of-modified-files-3.txt
::::::::::::::
::::::::::::::
cmorMDfixer-messages-1.log
::::::::::::::
Step 1: Before really applying the changes, create the olist for the --forceid case.

2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting metadata field parent_experiment_id to piControl in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr/
v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_child to 0.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr/v2024
0129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_parent to 149384.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr
/v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting tracking_id to hdl:21.14100/a498b0f4-2267-4343-ae95-a669473d2a24 for cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1
/Eyr/treeFrac/gr/v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Appending message about modification to the history attribute.
2024-10-15 15:22:39 INFO:cmorMDfixer.py: The latest applied cmorMDfixer version attribute is set to: v1.0
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting metadata field parent_experiment_id to piControl in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v202
40129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_child to 0.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v20240129
/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_parent to 149384.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v20
240129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Setting tracking_id to hdl:21.14100/565be458-1d8a-47e9-b2fb-b02edd592fab for cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1
/Eyr/cVeg/gr/v20240129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:39 INFO:cmorMDfixer.py: Appending message about modification to the history attribute.
2024-10-15 15:22:39 INFO:cmorMDfixer.py: The latest applied cmorMDfixer version attribute is set to: v1.0
::::::::::::::
cmorMDfixer-messages-2.log
::::::::::::::
Step 2: Apply the changes for the files listed in the olist for the --forceid case.

2024-10-15 15:22:40 WARNING:cmorMDfixer.py: Output file name list-of-modified-files-1.txt already exists, trying list-of-modified-files-2.txt
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting metadata field parent_experiment_id to piControl in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr/
v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_child to 0.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr/v2024
0129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_parent to 149384.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/treeFrac/gr
/v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting tracking_id to hdl:21.14100/9ab27663-cf8c-47c5-8fbd-18fc3f0aa3ef for cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1
/Eyr/treeFrac/gr/v20240129/treeFrac_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Appending message about modification to the history attribute.
2024-10-15 15:22:40 INFO:cmorMDfixer.py: The latest applied cmorMDfixer version attribute is set to: v1.0
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting metadata field parent_experiment_id to piControl in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v202
40129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_child to 0.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v20240129
/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting metadata field branch_time_in_parent to 149384.0D in cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1/Eyr/cVeg/gr/v20
240129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Setting tracking_id to hdl:21.14100/e06e46a7-bd74-42f5-a457-9c74b8e90fc2 for cmorMDfixer-test-data/test-set-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3/piControl/r1i1p1f1
/Eyr/cVeg/gr/v20240129/cVeg_Eyr_EC-Earth3_piControl_r1i1p1f1_gr_1850-1850.nc
2024-10-15 15:22:40 INFO:cmorMDfixer.py: Appending message about modification to the history attribute.
2024-10-15 15:22:40 INFO:cmorMDfixer.py: The latest applied cmorMDfixer version attribute is set to: v1.0
::::::::::::::
cmorMDfixer-messages-3.log
::::::::::::::
Step 3: Check whether there are still modifications detected which should have been applied already (possibly as a consequence of an earlier interruption).

2024-10-15 15:22:40 WARNING:cmorMDfixer.py: Output file name list-of-modified-files-1.txt already exists, trying list-of-modified-files-2.txt
2024-10-15 15:22:40 WARNING:cmorMDfixer.py: Output file name list-of-modified-files-2.txt already exists, trying list-of-modified-files-3.txt

So no empty files here? Let me know which hole I miss.

That's really strange, I ran the test case and it works as you describe, the list*txt files contain the files to be changed.
Then I try a "real" dataset, I have copied 5 years of a histroical run for OptimESM to a sandbox and would like to update experiment_id and activity_id. Here is my metdata-correction:
{
    "activity_id":                  "TIPMIP",
    "parent_experiment_id":         "my-own-experiment"
}

I then run
./cmorMDfixer-safe-mode-wrapper.sh 8 metadata-correction-cases/CD37-correction-0.json testbed/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon

which yields
 The changes are applied and agree with the preceding dry-run, so all seems fine (no interruption damage).

 All files in the entire dataset are correct after correcting, so ending successful!

 The versions.sh script detects the following versions in the final corrected dataset:
v20240622

Checking a few files in testbed/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon, I find that the metadata have been successfully updated. However, the list*txt files are all empty:
[sm_wyser@freja1 cmor-metadata-fixer]$ ls -l list*txt
-rw-r--r-- 1 sm_wyser sg_optimesm 0 Oct 15 15:44 list-of-modified-files-1.txt
-rw-r--r-- 1 sm_wyser sg_optimesm 0 Oct 15 15:44 list-of-modified-files-2.txt
-rw-r--r-- 1 sm_wyser sg_optimesm 0 Oct 15 15:44 list-of-modified-files-3.txt

I have put a copy of my example dataset on the Exporter: http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950

It seems the cmorMDfixer-*.log files contain correct information about files to be changed (in addition to other stuff)
Related to that something else: it seems that --forceid is always active which makes that every time cmorMDfixer is run it will create new IDs in the files and it will do so even if no other changes are made to a file. This is not very good, or? Files - for example variables -  that are not modified otherwise don't need new IDs.

I tried to download your data with:
wget -r -e robots=off http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950C
but I can't find any netcdf file in the download, which goes very fast, but only gives me index files?

The --forceid is set in the wrapper indeed and inherited from the wrapper use by I think Prashanth at the time. With cmorMDfixer you can choose a few ways how to use this. For data which has never been published before on the ESGF node in principle it doesn't matter how often the IDs are changes. Though for data diffing reasons you might not like it and prefer only chnaging IDs for data which is subject by a metadata modification.

I tried to download your data with: wget -r -e robots=off http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950C but I can't find any netcdf file in the download, which goes very fast, but only gives me index files?

Just a wild guess:  something with links (maybe also in the problem itself)?

I tried to download your data with: wget -r -e robots=off http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950C but I can't find any netcdf file in the download, which goes very fast, but only gives me index files?

Sorry, the link got an extra "C" at the end, here is the correct one: http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950

The --forceid is set in the wrapper indeed and inherited from the wrapper use by I think Prashanth at the time. With cmorMDfixer you can choose a few ways how to use this. For data which has never been published before on the ESGF node in principle it doesn't matter how often the IDs are changes. Though for data diffing reasons you might not like it and prefer only changing IDs for data which is subject by a metadata modification.

Exactly!

Of course we can have a look how we want this in the wrapper, if we are going to use the wrapper. The wrapper was of use when we handled large volumes of data and interruptions were a real potential issue.

Sure, that's something to have a look at when the rest is working.

I tried to download your data with: wget -r -e robots=off http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950C but I can't find any netcdf file in the download, which goes very fast, but only gives me index files?

Sorry, the link got an extra "C" at the end, here is the correct one: http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950

Well I am afraid the C was an typo of me in the post (removed it now), sorry for that confusion. But I still don't get any netcdf file with the correct address?

Weird. I'll check with NSC what's going on.

With:
wget -m -r -e robots=off http://exporter.nsc.liu.se/3da592671bda4d37b1576a8f225c1950
the download was successful.
But now with your data I run the default test and the test using your metadata modification. Both went fine, here! The metadata is changed and I get appropriate log and list files:
::::::::::::::
list-of-modified-files-1.txt
::::::::::::::
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185401-185412.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185301-185312.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185301-185312.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185401-185412.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185401-185412.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185301-185312.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
::::::::::::::
list-of-modified-files-2.txt
::::::::::::::
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185401-185412.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185301-185312.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185301-185312.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snd/gr/v20240622/snd_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185401-185412.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185401-185412.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185301-185312.nc
cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snw/gr/v20240622/snw_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
::::::::::::::
list-of-modified-files-3.txt
::::::::::::::

::::::::::::::
cmorMDfixer-messages-1.log
::::::::::::::
Step 1: Before really applying the changes, create the olist for the --forceid case.

2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting metadata field activity_id to TIPMIP in cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v
20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting metadata field parent_experiment_id to my-own-experiment in cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i
1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting tracking_id to hdl:21.14100/83406226-f77b-443f-a376-dbeba5f6ff8d for cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm
-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185201-185212.nc
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Appending message about modification to the history attribute.
2024-10-17 09:27:19 INFO:cmorMDfixer.py: The latest applied cmorMDfixer version attribute is set to: v1.0
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting metadata field activity_id to TIPMIP in cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v
20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting metadata field parent_experiment_id to my-own-experiment in cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i
1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting tracking_id to hdl:21.14100/2f57186f-b2f5-4c0a-85d1-d2dc567563fd for cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm
-hist/r5i1p1f1/LImon/snc/gr/v20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185101-185112.nc
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Appending message about modification to the history attribute.
2024-10-17 09:27:19 INFO:cmorMDfixer.py: The latest applied cmorMDfixer version attribute is set to: v1.0
2024-10-17 09:27:19 INFO:cmorMDfixer.py: Setting metadata field activity_id to TIPMIP in cmorMDfixer-test-data/test-set-optimesm-01/CMIP6/CMIP/EC-Earth-Consortium/EC-Earth3-ESM-1/esm-hist/r5i1p1f1/LImon/snc/gr/v
20240622/snc_LImon_EC-Earth3-ESM-1_esm-hist_r5i1p1f1_gr_185001-185012.nc
and so on ...

If I run with 1 proc I get
-rw-r--r-- 1 sm_wyser sg_optimesm 2400 Oct 17 11:19 list-of-modified-files-1.txt
-rw-r--r-- 1 sm_wyser sg_optimesm 2400 Oct 17 11:19 list-of-modified-files-2.txt
-rw-r--r-- 1 sm_wyser sg_optimesm    0 Oct 17 11:19 list-of-modified-files-3.txt

Looks fine. However, when I run the same example with 8 procs the list-of-modified-files-[1,2].txt files are empty:
-rw-r--r-- 1 sm_wyser sg_optimesm 0 Oct 17 11:20 list-of-modified-files-1.txt
-rw-r--r-- 1 sm_wyser sg_optimesm 0 Oct 17 11:20 list-of-modified-files-2.txt
-rw-r--r-- 1 sm_wyser sg_optimesm 0 Oct 17 11:20 list-of-modified-files-3.txt

So it's something with the parallelisation. That may not be a big deal, supposedly the cmorMDfixer is still doing what it is supposed to do, but can we really be sure about that? I'd really like to see the logfiles to check that the right files have been updated.

Ah, yes it probably is the parallelisation. I have to think how to solve that decently. I wonder whether the parallel part is really needed, it is anyway quick. But ok, let's see if this can be solved easily first.

The following easy basic test, shows the problem:
./cmorMDfixer.py --dry --verbose --forceid --olist --npp 2 metadata-correction-cases/knmi-metadata-corrections-piControl.json cmorMDfixer-test-data/test-set-01/

resulting in an empty list-of-modified-files-1.txt file.

I think I found the BUGfix. But do not fully understand yet why the code is as it is unless its a kind of typo. I first thought about a flush error (a lacking flush) but that part appeared to be ok. First going to test on the data now.

256b50f should fix this issue.


Overcomplicating Examples

Hi MS Team, The String method examples were really complicated, In future make simple example like readable and understandable by everyone. Most of the things in the document were understandable by expirienced ppl only i believe.

@Naveenpisher Thanks for taking the time to give us your feedback.
In the future, make sure to leave your comments on a specific page of documentation, so we can route your feedback to the correct team for improvement.
I've posted your request and we'll do our best to make improvements in future documentation releases.
Closing this issue for now, but feel free to continue commenting here as needed.


Update to RubyDNS 1.0.2

Fixes #120.
Update to RubyDNS 1.0.2.

Fixes #120. Celluloid 0.16.1 was released on Aug 8 which breaks RubyDNS.
RubyDNS 1.0.2 pinned to Celluloid 0.16.0. This will be fixed further in
a subsequent RubyDNS release.

Read through the thread and this makes sense, seems to fix people's stuff.
Super lame to change that behaviour in a .z release. :angry:


Allow formatting of C++ code

Added ability to indent properly the C++ generated code.
No 2 instructions on the same line.
I'm not sure if there are specific rules for formatting C++ code so the way I've formatted it might be subject to discussion.
See issue #833.
Size wise on my simple hello world, there is a size increase of about 5%. I did not benchmark the time spent in ilc nor to perform the C++ compilation.

I've put the formatted code currently being generated by this PR at: https://gist.github.com/manu-silicon/cea105dd7adb9ec0e7ba

LGTM modulo comment. Thanks for doing this!

I've fixed the debugging side, so when you do not specify -nolinenumbers it generates pretty much the same as before, even fixing a few debugging mishap with try/catch (to ensure we do not debug past the end of a routine try/catch clause).

I'm going to look at the Unix C++ compilation error.
I'm not sure why the Windows Release build failed.

The Windows build error looks like infrastructure problem.

It should be fixed now.

Could you please squash into a single commit?

done


Unable to query by geolocation

I want to fetch records using geolocation. is there any way to query using geolocation. please help me to resolve this issue.

The SDK has no out-of-the-box support for Geo Queries, it only provides a bridge to the official google/cloud-firestore library. I donMt know if Google's library has native support for them either.
Unfortunately I haven't worked with GeoPoints in Firestore yet, so I can't help you further.
Perhaps this Stack Overflow Link can give you a hint to the right direction: https://stackoverflow.com/questions/58791529/read-the-firestore-geopoint-object-in-php
Also, if you Search Gooogle for "GeoFire", you'll find implementations for GeoQueries, mostly for other programming languages, but perhaps you'll find something that can help.
I appreciate if you share your findings in further comments, so that others facing the same challenge can benefit from them 🤞🙏

is there any solution for this issue?


Chairs that aren't centered to the grid are impossible to unbuckle from without help

"You can't reach that", it says.
Buckling has returned for vengeance.

Tested and can't replicate. Closing.


Don't show report links on own content

In order to keep the menus lean, there should only be necessary options included. On your own content it's not really necessary to have a report button for yourself, because no one would report themselves.

Fixed, thank you!


NPM?

I feel bad to say that I bounced off this lib for three reasons: Its perfect in concept, but lacks NPM support as far as I can tell. I started porting it myself but the external dependencies became tedious to also port, and lastly, its not typed anywhere for typescript which would also be helpful.
This is your library and you can 100% run it however you want, but I thought the feedback was warranted if I decided not to use it.
Thank you for your work, this is a well put together library. Just a slight mismatch for me.

lacks NPM support as far as I can tell

I'm not opposed to npm support.  I haven't published there to iterate rapidly on concepts without worrying about supporting a stable interface (yet).  I expect at some point an npm package(s) will arrive. Probably sooner with help. ;)

I started porting it myself but the external dependencies became tedious to also port

Hmm, which external deps were tedious? There really aren't many; most of them are already npm packages, except for a couple tiny functions:  https://github.com/mreinstein/collision-2d/blob/main/src/deps.js

its not typed anywhere for typescript

Also not opposed to typescript support, though I really have no interest in it personally whatsoever.  I will happily support any autogenerating bits to keep the ts enthusiasts happy though!

This is your library and you can 100% run it however you want, but I thought the feedback was warranted if I decided not to use it.

Thank you for your feedback!  I wish you had engaged earlier instead of waiting until feeling burned out or frustrated with this module.  We could have probably worked together on these issues.  Waiting that long and then giving me a failing grade on my module is a bit de-motivating. :(

Thank you for your work, this is a well put together library. Just a slight mismatch for me.

Thank you for taking time to write, even if it is quite late in the process.  If you change your mind and want to tackle any of the above issues, PRs welcome! 💪

I'm approaching the time to publish an npm module for this now.
 I appreciate you doing this. For me personally, I will not have a chance to review in the short term, but I will gladly implement it and see how it works later in the month.

getting married 11/8

Ay, congrats!

see how it works later in the month

Sounds good! I've published this as a module because node 22 has killed it's experimental http imports, and I want this stuff to be isomorphic.
I'm not sure what you're building (a game, or sim, etc.)  But I'm gradually building out an entire modular data oriented 2D framework for these things. The other exposed modules are visible here: https://www.npmjs.com/org/footgun


Kim CNN: Tokenizer

Replace this with the PTB tokenizer.

Where should I put stanford-postagger.jar (NLTK)?

@daemon I was supposed to close this issue earlier since we found out that the datasets used by KimCNN (at least TREC and SST) are PTB tokenized. If we are going to extend KimCNN to other datasets, we would have to separately tokenize the text. I suggest we add a link in the README pointing to stanford-postagger.jar.


Fixing issue #3390

A transport op indicating a cancellation can be sent to the auth client
filter. In this case, the code should not assert that a context is not
null on this op.

@nicolasnoble may want to have a look as well if this is time sensitive for the 0.11.1 release.


add youtube iframe to session 2023-04-21

Feel free to commit these kinds of changes directly to the main branch.


JDBC driver consumes 100% CPU on a core since 325 release

All that time is being spent in Lio/prestosql/jdbc/PrestoResultSet$AsyncIterator;::hasNext
https://github.com/prestosql/presto/blob/334/presto-jdbc/src/main/java/io/prestosql/jdbc/PrestoResultSet.java#L1815
https://github.com/prestosql/presto/pull/965#discussion_r330199981 has more details

Related Slack thread is https://prestosql.slack.com/archives/CGB0QHWSW/p1591256731318900

cc @ShashwatArghode

Getting this right is pretty tricky. In addition to the current busy waiting in hasNext() I have a hard time convincing myself that we're not leaving threads hanging around (for example when the buffer is filled up, and nobody ever calls next/hasNext.
Maybe a simpler (slightly less performant) solution is to simply do pre-fetching. When a caller calls hasNext/next, we read a few more results from the passed iterator and the next calls would be served from the prefetch buffer.
We won't get the nice interleave of fetching and reading, but it'd be simpler and easier to reason about.

But, I do not know what part is slow about the fetching. If it's the lazy fetching one-by-one, then prefetching should be good enough. Perhaps someone can shine some light on that.
Prefetching itself is trivial (untested):
    private static class PrefetchIterator<T>
            implements Iterator<T>
    {
        private static final int MAX_QUEUED_ROWS = 10_000;

        private LinkedList<T> rowQueue = new LinkedList<>();
        private Iterator<T> source;

        public PrefetchIterator(Iterator<T> dataIterator)
        {
            requireNonNull(dataIterator, "dataIterator is null");
            this.source = dataIterator;
        }

        @Override
        public boolean hasNext()
        {
            if (rowQueue.isEmpty()) {
                for (int i = 0; i < MAX_QUEUED_ROWS && source.hasNext(); i++) {
                    rowQueue.add(source.next());
                }
            }
            return !rowQueue.isEmpty();
        }

        @Override
        public T next()
        {
            return rowQueue.poll();
        }
    }

FWIW, as a test I removed the AsyncIterator code; with everything on the localhost I see no slowdown.
With client and server on localhost we obviously have no networking latency. My data comes from Hive table (on a local HDFS)
For kicks I introduced a 10ms delay into my LOCAL network and still see no measurable difference in performance with or without the AsyncIterator.
So my question: What problem did we try to solve with #965 ?

Network latency between server and JDBC client? (so that we can interleave the server iteration with client retrieval)
Slow iterator for some of the connectors?

I think that #965 was not sufficiently motivated... At the very least I do not get what it is trying to achieve. Perhaps we should remove that code.

@lhofhansl I remember seeing a benchmark associated with the original PR, but I can't seem to find it now. @ShashwatArghode do you remember the use case that motivated this and the benchmark numbers?

Thanks @findepi for cc-ing.
@electrum This was a general improvement over the previous Presto code and I think was asked by users as well. The benchmark was performed on an actual Presto cluster on HDFS back then and this is the link.
I was looking into possible solutions
Approach 1:
downloadSpeed_END_TOKEN.txt is the patch in accordance with your #965 comment and also followed some of #2022
I think this patch will take care of queries that are erroring out but, we will have the same problem with slow running queries.
downloadSpeed_END_TOKEN.txt
Approach 2:
downloadSpeed_Wait:Notify.txt patch makes the hasNext() operation blocking. This will solve the busy loop problem but, this patch uses Thread.getState() and as per documentation this method is designed to be used for monitoring and not for synchronization control.
downloadSpeed_Wait:Notify.txt
@electrum @findepi Do you think any of this approach makes sense or can you suggest any other approach which will help with this?

@ShashwatArghode Using Thread.getState() is not the right way, as you noticed. The "end token" approach seems like a good way. What do you mean about "same problem with slow running queries"?
I think part of the problem is that we don't have a reproducible benchmark. @ShashwatArghode would you be able to reproduce your original benchmark results using the TPC-H connector?
@lhofhansl do you want to put a PR with your change? Fixing the 100% CPU problem seems like the most important thing right now. Can you try benchmarking with TPC-H as well? It would be helpful to see what is different between the scenarios.

@ShashwatArghode I looked into the token approach as well. It complex in the following scenario: The AsynchIterator filled it's blocking queue, the parent thread is interrupted, and the client is not reading from the iterator. In that case I could not convince myself on a good solution to guarantee the thread not hanging around indefinitely.
@r0hini I tried with 128m records reading in batches of 10k. But all on a single machine so that I can debug. I believe you that it was an improvement :)  Just figuring out what the problem is that we were trying to fix. Perhaps I do not see the effect.
Can you describe what exactly the AsyncIterator fixes? Is the problem the network RTT between the server and the JDBC client?
@electrum My change simply removes the AsyncIterator, it leaves everything else of #965 in place. I wanted to add prefetching, but then found that I cannot reproduce the problem even with artificially induced network latency.

But, I do not know what part is slow about the fetching. If it's the lazy fetching one-by-one, then prefetching should be good enough.
Since using separate threads for fetch and writing worked out really well for the CLI, we just went with the same approach for JDBC as well. Network latency wasn't the main reason even though it could be a factor if users were connecting via VPN and have poor wifi. Most of the time, it was the jdbc clients themselves being slow in processing a record and some custom clients were fetching using Presto and updating some KV stores. Prefetching solution you have proposed might also work in that case and I am good with that.

Thanks @r0hini just asking because if the problem is the source iterator itself being slow - and hence we wanted it to fill a queue while the JDBC client is doing its processing - then prefetching might not be effective since it would still be done with the JDBC client waiting for the prefetching.

@lhofhansl, you mentioned that you are not seeing any difference at all and thinking more on that, it does not sound right to me because the parallelization of fetching and parsing json in parallel should make some difference. Does you testing just loop through rs.next() without doing anything with the records or it at least has a print statement of each record? Can you give details of your benchmarking test, so that we can try reproducing?

@r0hini Hmm... I only get a value of the resultset every 10.000th call to next(). I'll change it to retrieve a value each time.
(And for the record, I'm not saying the parallelization is not needed, I'm trying to figure out the scenario, in order to figure out whether it's possible to simplify the logic and avoid the busy wait. :) )

Ok... Now when I introduce a 100ms (!) network delay I see about 25% improvement with the change.
I.e. ~1100ms to ~1400ms for 100k rows.
(again with 100k rows each time, and an artificial network latency of 100ms)
That is without and prefetching, just removing the AsyncIterator. I just cannot find a scenario that would justify the extra busy waiting.

Intent was to speed up the total response time and since it was single threaded with laptops to servers hosting BI tools having more cores available, we decided to use separate threads for speeding up. One for fetch (both network time and compute time of parsing json http response->objects) and another for format/print (mainly compute time but possible that the custom jdbc user code involves some network time as well).  Can you also try tables with datatypes like timestamp and doing rs.getTimeStamp(). I think you will see more difference due to increased compute.

I just cannot find a scenario that would justify the extra busy waiting.
Not disagreeing on that. We should definitely get rid of the busy wait and also address possible leaking of threads that you raised. If it takes more time to solve it right or come up with a better solution as Presto release cycles are very short, we can revert it for the next release and fix it again after that. If pre-fetching works we can go with that, but I now doubt that it will be as good as using more threads.

Thanks @r0hini  I'll try with timestamps too.
We can wait for two conditions with a traditional Java monitor (Java synchronized). Each time we add something to queue or when the Future is done we notify(). On the other thread we just check the condition and wait() in a loop.  Problem is we'd have to enter and leave the monitor a lot (on the "sending" side, which at least need to perform a memory barrier, and that might negate the speed improvement). Or use a Semaphore (I assume that would have the same overhead as a monitor - mostly a memory barrier).

I have a version with a counting Semaphore that does not busy looping. But all the synchronization overhead seems to outweigh the benefit. I see it taking about 1450ms/100.000 iterations. (simple getInt() still)

Here's the change with the Semaphore... Pretty simple, just counts the number of expected rows in a Semaphore, and uses that to block.
diff --git a/presto-jdbc/src/main/java/io/prestosql/jdbc/PrestoResultSet.java b/presto-jdbc/src/main/java/io/prestosql/jdbc/PrestoResultSet.java
index cc874c41a4..c93ff4f22d 100644
--- a/presto-jdbc/src/main/java/io/prestosql/jdbc/PrestoResultSet.java
+++ b/presto-jdbc/src/main/java/io/prestosql/jdbc/PrestoResultSet.java
@@ -64,6 +64,7 @@ import java.util.concurrent.BlockingQueue;
 import java.util.concurrent.CompletableFuture;
 import java.util.concurrent.ExecutionException;
 import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Semaphore;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicReference;
 import java.util.function.Consumer;
@@ -1801,6 +1802,7 @@ public class PrestoResultSet
         private final StatementClient client;
         private final BlockingQueue<T> rowQueue = new ArrayBlockingQueue<>(MAX_QUEUED_ROWS);
         private final CompletableFuture<Void> future;
+        private final Semaphore s = new Semaphore(0);
         private Thread parent;

         public AsyncIterator(Iterator<T> dataIterator, StatementClient client, Thread parent)
@@ -1808,17 +1810,17 @@ public class PrestoResultSet
             requireNonNull(dataIterator, "dataIterator is null");
             this.parent = parent;
             this.client = client;
-            this.future = CompletableFuture.supplyAsync(() -> {
+            this.future = CompletableFuture.runAsync(() -> {
                 while (dataIterator.hasNext()) {
                     try {
                         rowQueue.put(dataIterator.next());
+                        s.release();
                     }
                     catch (InterruptedException e) {
                         interrupt(e);
                     }
                 }
-                return null;
-            }, executorService);
+            }, executorService).whenComplete((result, ex) -> { s.release(); });
         }

         public void cancel()
@@ -1836,14 +1838,14 @@ public class PrestoResultSet
         @Override
         public boolean hasNext()
         {
-            if (!rowQueue.isEmpty()) {
-                return true;
+            try {
+                s.acquire();
             }
-            while (rowQueue.isEmpty() && !future.isDone()) {
-                // making sure rowQueue has some records to process or return false
-                if (parent.isInterrupted()) {
-                    interrupt(new InterruptedException("parent thread interrupted"));
-                }
+            catch (InterruptedException e) {
+                interrupt(e);
+            }
+            if (parent.isInterrupted()) {
+                interrupt(new InterruptedException("parent thread interrupted"));
             }
             if (future.isCompletedExceptionally()) {
                 try {

@electrum By slow running queries, I meant the datastore which is inherently slow in providing data to the iterator. In this case, we will busy loop until the data is available.
@lhofhansl, as @electrum suggested I used lineitem table from sf30 schema from the TPC-H catalog with single node (presto-coordinator + presto worker) locally. I did a select * on the table and just printed all values as a string on the console. (The tests were done without the end_token/wait_notify patch)
Total Rows: ~180M
Time-taken by new driver: 2595.97 seconds
Time-taken by the old driver: 3632.43 seconds
The results indicate that we are able to optimize the fetching of the query result with this approach, we need an efficient solution to avoid busy looping though.

I have a version with a counting Semaphore that does not busy looping. But all the synchronization overhead seems to outweigh the benefit. I see it taking about 1450ms/100.000 iterations. (simple getInt() still)

You can also run a similar benchmark to verify this.

@ShashwatArghode Any chance that you could try with the simple Semaphore change above? (I'll be a bit busy in the next few days.)
When either source iterator or the network is slow the extra synchronization overhead should be absolutely negligible.

If it takes more time to solve it right or come up with a better solution as Presto release cycles are very short, we can revert it for the next release and fix it again after that.

@r0hini  I think it is a good idea.
Would you like to do that?

@lhofhansl
I am a bit surprised additional synchronization removes all the benefit from parallel work.
It could be in the case where "client app" (consumer of results) does very little, but should not be the case when client app does more work.
This is not to say I reject you observations. I am a bit confused which scenario you were testing this against.
Also, if we have to reduce the overhead of synchronization, we could do hand over of batches, instead of individual rows, and then flatten this on the reading side
(this without additional synchronization).

@findepi I am too, and I am now questioning my testing. The Semaphore change is simple and easy to reason about... should be good. Perhaps I'll do a PR...?

Posted some number over in #4100


Fix logic errors and races in initial invite and disconnection

This fixes false-negative presence when receiving an invite, and
false-positive presence after a friend has disconnected.

:+1:


500 Internal Server Error on CloudFormation

I just spent three days going back and forth between AWS Support to try and get to the bottom of a 500 internal server error on CloudFormation when trying to setup Lambda
An error occurred and the request cannot be processed. (Service: AWSLambdaInternal; Status Code: 500; Error Code: ServiceException; Request ID: xxxxxx)
Stack ID(s) / Stack ARN: xxxxxxxxx

I wanted to create this topic just to try and help people by indexing the problem and what worked for me. My issue was on the layer ARN region. Instead of - arn:aws:lambda:eu-west-1:209497400698:layer:console:1 I had a typo on the region which led to - arn:aws:lambda:eu-wast-1:209497400698:layer:console:1 and as of today, this typo is not being validated by CloudFormation, which lead to a nasty internal server error that took a lot of time even for AWS Support Engineers to find.
Hope this help people avoid wasting time.

:+1:, I've have a problem of the kind at some point (using the wrong region) and say as you it was very cryptic to resolve :/
Hopefully they improve that both in CloudFormation and sam local…

For anyone reading this and still having the same problem, here is my recommendation: check if there is any parameter with a null or empty value. In my case, I was using a parameter to construct the principal for a Lambda permission, but I forgot to pass the value to the parameter and ran into this obscure issue.

For us it was typo in ARN:
"layers": [
          "arn:awscloud:lambda:us-east-1:XXXX:layer:XXX_nodejs:1"
        ],


ol.proj.transform unhandled exception

it appears there is an error when converting geographics to MGA/UTM coords
ol.proj.transform([151.60856227969708, -32.89173695452235], 'EPSG:4326', 'EPSG:32756');
ol.proj.transform([369853.293895506, 6359901.25733753], 'EPSG:28356', 'EPSG:900913')
ol.proj.transform([369853.293895506, 6359901.25733753], 'EPSG:32756', 'EPSG:900913')
Crashing is:
Uncaught TypeError: Cannot read property 'getCode' of undefined.
ol-debug.js: 19280  version 3.4

Please report issues with ol-debug.js see: http://openlayers.org/ol3-workshop/basics/resources.html#reporting-issues

sorry bartvde, a noob mistake, I forgot I was using the non debug version.  I've now rerun it using the debug libraries.

Looks like transforms between projections you use are not defined. The easiest way to get them defined is to load the Proj4js library and the CRS definitions for EPSG:28356 and EPSG:32756, which you can get e.g. from http://epsg.io/.

This not the case.  My work around was to incorporate the OpenLayers v2.x library which needed the Proj4js library, as well asn conversion string.
Below are the relevant chunks of code.

Proj4js.defs["EPSG:27700"] = "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +datum=OSGB36 +units=m +no_defs";
Proj4js.defs["EPSG:4326"] = "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs";
Proj4js.defs["EPSG:900913"] = "+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +no_defs";
Proj4js.defs["EPSG:28356"] = "+proj=utm +zone=56 +south +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs ";
var wgs82 = new OpenLayers.Projection("EPSG:4326");
var mga56 = new OpenLayers.Projection("EPSG:28356");
var center = ol.extent.getCenter(extent);
var point = new OpenLayers.LonLat(center[0], center[1]).transform(mga56, new OpenLayers.Projection("EPSG:900913"));
view.setCenter([point.lon, point.lat]);

You need to use proj4js 2.x with OpenLayers 3, not 1.x.

thanks guy.  i'll look into it.


Under dnd5e version 3.x, CSS templates not applied.

When using this mod on the 3.x release of dnd5e, the chat messages are created without their color-coding.  Data provided in chat cards is presented correctly:

You can temporarily restore the mod by modifying styles/dnd5e-character-monitor.css in the mod directory. You only need to add the !important description to the styles in the following pictures. I hope this can be helpful to you.

@jessev14 can you patch this?


[DOCS] Update the mentoring projects

have done some work on the project pages, it would be good if someone else could proof read it and add anything they think would be good.

I would rather remove the ones which are already taken IMO.

Which ones are any taken? I was going to have a completed list once we have any, but Im not aware of any that have been assigned yet

Fair enough, getting ahead of myself a bit there 🙂.


[Bug 2.3] Classifier not correctly supported in lock-snaphots

I have a dependency of the following form in the dependency management section of my BOM.
<dependency>                 <groupId>a.b.c</groupId>                 <artifactId>my-artifact</artifactId>                 <version>2.1.3-SNAPSHOT</version>                 <classifier>resources</classifier>                 <type>tar.gz</type>             </dependency>
I then try to run the following maven command: 'mvn clean org.codehaus.mojo:versions-maven-plugin:2.3-SNAPSHOT:lock-snapshots'
I get an error when the plugin tries to contact the repository saying he cannot find the artifact. The artifact is however present in the repository.
EDIT: I get exactly the same error when it is a standard dependency.
stack.txt

Can you provide a full working example on github

Publish any artifact with a classifier in your local repository then try to lock snapshots you should get the error

So checked with current master. Works now without any issue. I will not close it until I have a working integration test(s) for this....


KAFKA-3931: Fix transient failures in pattern subscription tests

Full credit for figuring out the cause of these failures goes to @hachikuji.

@ijuma @guozhangwang I reverted part of the other commit and used a different fix. I could not reproduce the failures locally with this update. Not sure yet what made the previous code to act inconsistently, but I thought this might get the builds pass, and I'll try to figure that out in the meantime.

cc @ewencp @hachikuji

The new fix that I mentioned in my earlier comment adds minimal changes to the original SubscriptionState.java.
IMO, the implementation is more readable in what I proposed (here) in the merged pull request, but that solution seems to bring some inconsistency (causing intermittent test failures) that I still have no explanation for.

@vahidhashemian, have you been able to figure out what is the difference in behaviour between this and the code that is currently in trunk?

@ijuma I have just started looking at the logs between the two. I'll share if I find anything interesting.

Thanks.

Reverting the change may be a little extreme until we understand the cause of the failure (I actually preferred the use of the enum). I haven't been able to reproduce the failure locally in order to debug, but I can try on jenkins.

@hachikuji, yeah, that's why no-one merged this yet. However, the failures happen often enough that it's affecting other PRs. If we can't get to the bottom of it soon, we may have to either revert the original PR or merge this one.

The only way I have been able to get either of the three pattern tests to fail has been running them together. I disable all other tests and let these three to be the remaining active tests. Running single tests has not resulted in  a failure for me so far.

@vahidhashemian I think I see what's going on. It looks like there's a race condition with the initial metadata fetch and the initial rebalance. We don't actually ensure that the metadata is fresh before joining initially, but we request the metadata update. If that metadata update arrives while the rebalance is still pending (for example, when the join group is still inflight), then we will lose track of the fact that we need to rebalance again to reflect the change to the topic subscription. This seems to be a general problem that we haven't considered before. Any metadata update which changes the topic subscriptions which arrives with a rebalance in progress will essentially be ignored.

A quick fix which doesn't address the problem completely, but which should get the tests passing again would be to do something like this in ConsumerCoordinator.ensurePartitionAssignment():
        if (subscriptions.partitionsAutoAssigned()) {
            if (subscriptions.hasPatternSubscription())
                client.ensureFreshMetadata();
            ensureActiveGroup();
        }

@hachikuji Thanks a lot for the explanation and the suggested fix. I tested it a bit and could not reproduce the failures anymore. Should I update this PR with this fix? Or you'll create a new PR? Thanks.

@vahidhashemian I think it makes sense to do it here. I'll open another JIRA to make sure we solve the general problem, but I think this is a necessary part of the solution.

@hachikuji Sounds good. Then I'll go ahead and update this PR. Thanks.

FYI, here's a link to the JIRA: https://issues.apache.org/jira/browse/KAFKA-3949.

LGTM. Merged to trunk and 0.10.0 branch.


Support Posix Flags

Currently multiple flags can't be combined, for example:
./cmd -a -b arg can't be shortened to ./cmd -ab arg
I know a benefit of cdr/cli is that it has no external deps, but I think it would be a better UX if we used https://github.com/spf13/pflag to support posix flags.

Yes, I agree. Standard flag is really not good.


Implementation of BigNumber utility class

Description
Nevermined BigNumber utility class to abstract the details of the underlying bignumber library. The goal is to in the future be able to replace the underlying bignumber library without introducing breaking changes

removed bignumber.js

Is this PR related with an open issue?
Related to Issue #226
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

[x] Follows the code style of this project.
[ ] Tests Cover Changes
[x] Documentation

Funny gif

Looks good to me. Just 2 comments:

It's tested in conjunction with the rest of the integration tests, but maybe an independent unit test for the BigNumber class would be good
Not sure about extending a class with same name can be cause of conflicts/problems/confusion: BigNumber extends ethers.BigNumber


Cannot get output with Chez scheme

Hi,
I'm using Geiser to get output from org-babel code block. This function works fine with guile and racket, but I cannot get it works with chez scheme.
When I create a code block with this simple display code:
#+BEGIN_SRC scheme :results output
  (display "Test string")
#+END_SRC

The Geiser dbg buffer only shows this result when I was using Chez:

=> #<void>

Mean while, it show this when I was using Guile and Racket.
(display "Test string")

=> #<void>

Test string

Could you show me how to get the output from Chez REPL?
Thank you so much

we are moving the project to gitlab. so please, follow-up there as needed. sorry for the inconvenience!
http://gitlab.com/geiser/issues/248


Android-Java: 9.1 Media Player Example

Description

How media works in the android system
Controls

Domain
Android Dev (Java)
Type of Contribution
Documentation
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

/assign


Serialization for GPFlow objects

Feature request

It would be very beneficial for GPFlow objects to support keras-style serialization and deserialization. The implementation for this merely requires the addition of get_config and from_config methods to all objects, as well as deserialization functions for each submodule (e.g. we would deserialize a likelihood with likelihoods.deserialize(config), and get_weights and set_weights methods for gpflow.model objects which store their own internal data.
Motivation
It is frustrating having to define and retrain models from scratch between sessions; it would be very useful to be able to serialize and save models (with their trained values preserved either through get_config methods, which are sufficient to capture low-dimensional parameters, or through get_weights) for reinstantiation later.
Proposal
Describe the solution you would like
As above, implementation of get_config and from_config methods on all objects, as well as serialization functions for each submodule and get_weights and set_weights methods for models.
What alternatives have you considered?
Alternatives N/A
Are you willing to open a pull request? (We really appreciate contributions!)
Yes, I am willing to open a pull request.

Hi @wbeardall, this is indeed something we would like to see GPflow be able to do! In the past there had been some additional issues with saving models (resolved in GPflow 2.1). At this point, it'd be great if we could make serializing properly. I believe this is something we could do in a piece-wise fashion, it'd be amazing if you would contribute the first steps in this direction! Please let us know if you've got any questions as to how to proceed - join the GPflow slack (invite link is in README.md) if you'd like to discuss more directly:)

Hi ST, I currently have a (temporary) working implementation of a serialized GPFlow, which works by dynamically creating the configuration methods for objects and subclassing them. However, because it's a metaprogramming-esque implementation, it isn't really suitable as a format for merging with the release version of GPFlow. I would like to refactor it into a more stable (and readable!) implementation in the relatively near future, though, so I'll update when I get a chance.

Hi @wbeardall, please do keep us up to date on this. I'd be very happy to help you with getting this in. I've been looking at good serialisation of GPflow objects for a long time as well :)


CPU, memory and disk metrics cannot be displayed.

Describe the bug
CPU, memory and disk metrics cannot be displayed.
To Reproduce
Steps to reproduce the behavior:

Install version 2023.3.21002-latest.
Add my k8s cluster.
Click on My K8S Cluster--> Settings--> Metrics --> PROMETHEUS select  Prometheus Operator --> PROMETHEUS SERVICE ADDRESS select  monitoring/prometheus-k8s:9090 --> ESC（I am using kube-prometheus）
Metrics are not available due to missing or invalid Prometheus configuration. Open cluster settings
It doesn't work even after following lens/custom-prometheus.

Expected behavior
Metrics to show up.
Screenshots

Environment (please complete the following information):

Lens Version: 2023.3.21002-latest
K8s Version: v1.25.2
Prometheus Version: v2.41.0
Node-exporter: v1.5.0
Installation method (e.g. snap or AppImage in Linux):

Logs:
When you run the application executable from command line you will see some logging output. Please paste them here:
Your logs go here...

Kubeconfig:
Quite often the problems are caused by malformed kubeconfig which the application tries to load. Please share your kubeconfig, remember to remove any secret and sensitive information.
your kubeconfig here

Additional context
Add any other context about the problem here.

Same issue on v 2023.3.31330-latest, before updating it was working

Following, same issue here, Any workarounds / dates when it will be fixed?

+1

me too

Thanks for the details. Though I will close this as a duplicate of #7264

I have the same problem，version 2022.11.251411-latest,I try to update to  v2023.3.71735-latest (2023/3/8), it does not work still.

我有同样的问题，版本2022.11.251411-latest，我尝试更新到v2023.3.71735-latest (2023/3/8)，还是不行。

and I have tried all the promethes configure methods


Unable to extract ZIP file on OS X

Hi, for some reason OS X's Archive Utility app has some trouble working with the ZIP file for OS X (Darwin). When I attempt to extract it, this is the error dialog I get:

Moving the file to another directory doesn't solve the issue (so it isn't permissions), and I'm able to extract it successfully using third party apps such as The Unarchiver or BetterZip (so it isn't corruption).

Thanks for the input! I'm updating the build with a new zip using the native tool

@bkromhout New versions directly provide dmg/deb/exe binaries without the zip wrappers, closing the issue.


11/13/2018 04:50 : Create Pull Request for branch prTesting_2018111304 -> master

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details
Dependent Files

split/updateTestCon.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.