Add Environment Variable DynamicConfigProvider

As #9351 described, DynamicConfigProvider should replace PasswordProvider. PasswordProvider has an implementation named EnvironmentVariablePasswordProvider, which allow users to read config from environment variable. This PR provides a same implementation for DynamicConfigProvider. There is an example for its usage:
"DynamicConfigProvider": {
  "type": "environment",
  "variables":
         {
            "userNameVariable": "MY_USER_NAME_VAR",
            "passwordVariable": "MY_PASSWORD_VAR"
         }
}

This PR has:

[ ] been self-reviewed.

[ ] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[ ] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[ ] added or updated version, license, or notice information in licenses.yaml
[ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[ ] been tested in a test Druid cluster.

@bananaaggle thanks for the quick response. The code change LGTM, but there are 2 things missing here, one is documentation and another is tests. For documentation, I think you can add some in docs/operations/dynamic-config-provider.md. For tests, you can add some unit tests in EnvironmentVariableDynamicConfigProviderTest that should run with a proper set of environment variables. To let Travis CI run those new tests successfully, you will need to add those environment variables in .travis.yaml too. I can help you with adding these, let me know if you have questions.

@bananaaggle thanks for the quick response. The code change LGTM, but there are 2 things missing here, one is documentation and another is tests. For documentation, I think you can add some in docs/operations/dynamic-config-provider.md. For tests, you can add some unit tests in EnvironmentVariableDynamicConfigProviderTest that should run with a proper set of environment variables. To let Travis CI run those new tests successfully, you will need to add those environment variables in .travis.yaml too. I can help you with adding these, let me know if you have questions.

Document added. I'll add more tests in week.

@bananaaggle thanks for the quick response. The code change LGTM, but there are 2 things missing here, one is documentation and another is tests. For documentation, I think you can add some in docs/operations/dynamic-config-provider.md. For tests, you can add some unit tests in EnvironmentVariableDynamicConfigProviderTest that should run with a proper set of environment variables. To let Travis CI run those new tests successfully, you will need to add those environment variables in .travis.yaml too. I can help you with adding these, let me know if you have questions.

Hi, @jihoonson. I add unit test for getConfig(). To set environment variable, I find a method named setEnv(Map<String, String> newenv) in stackoverflow which can change jvm environment variable by reflection. Do you think it is a proper solution? Is necessary to add environment variables in .travis.yaml? If add environment variables in .travis.yaml, this unit test will work in travis, but users can not pass it without travis. What unit tests you think is necessary to be added in next step?

Hi @jihoonson! I've changed this unit test but CI tests failed. It seems caused by travis error, can you help me rerun it? And is there something in code needed to change? Thanks!

@bananaaggle sorry, I forgot about this PR. I restarted the timed-out tests. Will finish my review today.

LGTM. Thanks @bananaaggle!

Thank you very much for review! In 11389, I make a design to replace PasswordProvider, but I'm not sure this design is proper or not. Can you help review it? If design is proper, I will involve more classes which use PasswordProvider in that PR. I also comment about it in 9351.


DPDK: check supported OS versions in a single place

Refactor: dpdk suite isn't supported on all platforms, refactor old checks into a single function check for DPDK support and skip others.

Please fix the failure and merge into one commit.


TRANSFER_FAILED

I am getting transfer_failed error , execution reverted when bot buy.

I see this errors in bscscan txid

Token i want to buy has tax for buy and sell, can this be cousing error, where can we adjust slipage in code?

Did some research, you need to use swapExactTokensForTokensSupportingFeeOnTransferTokens function in code to be able to buy/sell tokens with tax.

Gas was too low. Working!


How can I train maskformer with vit-comer?

Wonder if you could upload the config of maskformer with vit-comer? Thanks a lot.

We upload the config of maskformer with vit-comer based on COCO-Stuff-164K.


Add area support to group service schemas

Description:
Part of https://github.com/home-assistant/home-assistant/issues/25155.
Related issue (if applicable): fixes #
Pull request with documentation for home-assistant.io (if applicable): home-assistant/home-assistant.io#<home-assistant.io PR number goes here>
Example entry for configuration.yaml (if applicable):

Checklist:

[ ] The code change is tested and works locally.
[ ] Local tests pass with tox. Your PR cannot be merged unless tests pass
[ ] There is no commented out code in this PR.
[ ] I have followed the development checklist

If user exposed functionality or configuration variables are added/changed:

[ ] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] The manifest file has all fields filled out correctly. Update and include derived files by running python3 -m script.hassfest.
[ ] New or updated dependencies have been added to requirements_all.txt by running python3 -m script.gen_requirements_all.
[ ] Untested files have been added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Can someone check my assumption that SET_SERVICE_SCHEMA and REMOVE_SERVICE_SCHEMA don't require this?

Can someone check my assumption that SET_SERVICE_SCHEMA and REMOVE_SERVICE_SCHEMA don't require this?

Seems reasonable to me.  I can't think of a use-case for set... I could see a potential for remove though, but for consistency, I think it's OK for now.  We can always add it in the future if someone comes up with a use case.


S3 bucket goes missing while using pytest fixtures

Hello, I am experiencing some weird behavior when using moto along with pytest fixtures.
Any ideas why the bucket would seem to not exist?

Your use of with mock_s3(): in two different places is causing two separate mock environments to be created and destroyed.  The bucket you create is only available within the context of the s3_setup function.  Try removing both with statements and decorating only the test_moto_s3 method with @mock_s3.

I believe @bpandola is correct. Feel free to reply if that doesn't fix the issue.

Sorry for not following up before, but switching to using one @mock_s3 call did not fix the issue. The context does not seem to be applied to fixtures. I worked around it by not using fixtures for AWS-related things in my test suite; obviously not ideal but I can make it work for now.

@spulec Unfortunately, that's not correct. pytest will look at the test function's argument list (names) and create fixture instances before even calling the test function. So dropping @mock_s3 from the fixture call will attempt to create a live bucket.
What pattern are you using/suggesting to create fixtures (buckets or objects that need to exist for multiple unit test)?

I don't use pytest fixtures personally so I can't speak to it.
I either use class-based tests and setup* methods or I create a decorator that does the setup for me.

For anyone else landing here:
You probably want to use this as a fixture.
In my conftesy.py I have this:
@pytest.yield_fixture
def s3():
    with mock_s3():
        s3 = boto3.client("s3")
        yield s3

And in my tests I add an s3 parameter and everything works as expected
def test_something_with(s3):
    pass

Here's another approach, using with in the fixture (inspired by the comments in https://github.com/spulec/moto/issues/1568):
import boto3
import pytest
from moto import mock_s3

def function_under_test():
    client = boto3.client("s3")
    obj = client.get_object(Bucket="test-bucket", Key="test-key.txt")
    return obj["Body"].read().decode("utf-8")

@pytest.fixture
def s3_mock():
    with mock_s3():
        conn = boto3.resource('s3', region_name='us-east-1')
        bucket = conn.create_bucket(Bucket="test-bucket")
        bucket.put_object(Body="hello from moto", Key="test-key.txt")
        yield

def test_function(s3_mock):
    assert function_under_test() == "hello from moto"


Kotlin Coroutines Support

Add Kotlin Coroutines specific support.
Kotlin Coroutines support as alternative type of the CompletableFuture in DataLoader
Similar to the existing ReactorDataLoader(internal class in the Spring GraphQl) etc.

Create CoroutinesBatchLoader and CoroutinesMappedBatchLoader variants.
Create a Kotlin Coroutines variant of DataLoader.

All Controller(Data Fetchers) fun supports Kotlin Coroutines
Currently I encountered an issue when returning a Flow in a query data fetching. For example,
Define the following Graphql schema.
type Query {
    allPosts: [Post!]!
}

When executing the following query.
@QueryMapping
fun allPosts(): Flow<Post> = postService.allPosts()

I got the exception like this.
error message: Can't resolve value (/allPosts) : type mismatch error, expected type LIST

Reproducing the issue: https://github.com/hantsy/spring-graphql-sample/blob/master/spring-graphql-rsocket-kotlin-co
Support Flow as Subscription return type
In a Kotlin Coroutines project, I would like use a Flow instead of Flux/Publisher as Subscription return type, that make my codes look smooth, no mixes of Reactor API and Kotlin Coroutines API.
val flow=MutableSharedFlow<Event>()

//emit event
flow.emit(event)

//subscribe in a Subscription
fun onEventOccured() = flow (instead of a Flux/Publisher)

Currently I have to convert it back to a Reactor Flux to make it work.
override fun commentAdded(): Flux<Comment> = flow.asFlux()

The example codes: https://github.com/hantsy/spring-graphql-sample/blob/master/spring-graphql-rsocket-kotlin-co/src/main/kotlin/com/example/demo/Services.kt#L92
Spring GraphQl has built-in ReactorBatchLoader internally. I think it is possible to use the same approach to get Flow support as a Subscription type.
Kotlin Coroutine variant of DataFetcherExceptionResolver
Provides a variant to return suspend or Flow instead of the Reactor APIs.

I'm not sure that's how the implementation would work out. In general, it is more helpful to describe the ask rather than a solution. Also this overlaps with #406 and #407 while the title here is broad enough to cover all three.
If you could please, edit the description to describe all the places where you'd like to see Kotlin support. I'm gong to close the other two issues as duplicates. If we need to create sub-tasks we will, but for the requirements I see no reason to have multiple issues.

OK, I listed all my thought and issues when creating the example project: https://github.com/hantsy/spring-graphql-sample/blob/master/spring-graphql-rsocket-kotlin-co

I am also facing Can't resolve value (/listItems) : type mismatch error, expected type LIST got class kotlinx.coroutines.reactive.PublisherAsFlow error
any workaround ?
https://github.com/xmlking/micro-apps/tree/main/services/spring-graphql-r2dbc

@xmlking  Call yourflow.toList() to convert it to a List now.

checking graphql-kotlin-spring-server spring project,   looks like they support Flow but this project compatible with official spring graphql project.
https://opensource.expediagroup.com/graphql-kotlin/docs/server/spring-server/spring-subscriptions

@xmlking I have tried it in my example project.


Support WEBP? [add findTextureSource()]

Hi, I was running into an issue where webp files would not load due to texture.source being undefined. The source, in case of a webp file seems to be under 'texture.extensions.EXT_texture_webp.source'. To address this, I wrote a small function that searches for the 'source' key. I do not know if this is the 'right' fix, but it did fix my problem of webp's not loading.

@MatsErdkamp Did this change not fix your issue?

It did. Found a bug so closed it, but forgot I was working with a local patch-package version. Will reopen


Routes are not learned through OpenVPN site-to-site configuration

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

[X] I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
[X] I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug
While setting up a site-to-site VPN utilizing OpenVPN, routes were not being properly learned, requiring manual addition through the CLI and/or WebGUI.
To Reproduce
Steps to reproduce the behavior:

Configure OpenVPN on relevant OPNsense instances as per https://docs.opnsense.org/manual/how-tos/sslvpn_instance_s2s.html
Visit VPN -> OpenVPN -> Connection Status on either instance
Observe that either end report ERROR: FreeBSD route add command failed: external program exited with error status: 1
Visit System -> Routes -> Configuration
Observe that the only routes learned are the OpenVPN tunnel network, not any of the site networks.
Upon shutdown of OpenVPN, observe that either end report ERROR: FreeBSD route delete command failed: external program exited with error status: 1

Expected behavior
Routes are learned successfully through proper configuration of the OpenVPN server
Describe alternatives you considered
Accessing the CLI on both routers and running route add $network_in_cidr_format $destination_ovpn_ip_address completed successfully and correctly, after deleting using the relevant command and using the web GUI it added successfully as well, persisting after reboots
Screenshots
Server configuration:

Server routes learned (Via VPN -> OpenVPN -> Connection Status -> Routes)

Server routes learned (Via System -> Routes -> Status)

Client configuration:

Client routes learned (Via VPN -> OpenVPN -> Connection Status -> Routes)

Client routes learned (Via System -> Routes -> Status)

Relevant log files
Server OpenVPN logs:
2024-02-22T09:41:39-06:00	Warning	openvpn_server1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:41:39-06:00	Warning	openvpn_server1	 WARNING: --keepalive option is missing from server config
2024-02-22T09:40:29-06:00	Error	openvpn_server1	 event_wait : Interrupted system call (fd=-1,code=4)
2024-02-22T09:34:58-06:00	Warning	openvpn_server1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:34:58-06:00	Warning	openvpn_server1	 WARNING: --keepalive option is missing from server config
2024-02-22T09:34:55-06:00	Error	openvpn_server1	 event_wait : Interrupted system call (fd=-1,code=4)
2024-02-22T09:34:40-06:00	Warning	openvpn_server1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:34:40-06:00	Warning	openvpn_server1	 WARNING: --keepalive option is missing from server config

Client OpenVPN logs:
2024-02-22T09:41:54-06:00	Warning	openvpn_client1	 ERROR: FreeBSD route add command failed: external program exited with error status: 1
2024-02-22T09:41:54-06:00	Warning	openvpn_client1	 ERROR: FreeBSD route add command failed: external program exited with error status: 1
2024-02-22T09:41:53-06:00	Warning	openvpn_client1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:41:53-06:00	Warning	openvpn_client1	 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
2024-02-22T09:39:36-06:00	Warning	openvpn_client1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:39:36-06:00	Warning	openvpn_client1	 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
2024-02-22T09:37:33-06:00	Warning	openvpn_client1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:37:33-06:00	Warning	openvpn_client1	 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
2024-02-22T09:35:18-06:00	Warning	openvpn_client1	 ERROR: FreeBSD route add command failed: external program exited with error status: 1
2024-02-22T09:35:16-06:00	Warning	openvpn_client1	 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
2024-02-22T09:35:16-06:00	Warning	openvpn_client1	 WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
2024-02-22T09:35:14-06:00	Warning	openvpn_client1	 ERROR: FreeBSD route add command failed: external program exited with error status: 1
2024-02-22T09:35:14-06:00	Warning	openvpn_client1	 ERROR: FreeBSD route delete command failed: external program exited with error status: 1

Note: Logs have been truncated due to their length. I am willing to provide full logs upon request.
Additional context
This has been performed in a virtual environment on VMware Workstation 16 Pro. Due to everything being virtualized and no sensitive information being used in these, I am willing to provide the configuration used on both the client and server endpoints to facilitate troubleshooting.
Environment
Software version used and hardware type if relevant, e.g.:
OPNsense 24.1.2_1 (amd64).
Intel Core i5-1135G7 (each VM has 4 cores)
NICs: Intel 82545EM (Emulated through VMware)
OpenVPN version: 2.6.9
Server network: <IP_ADDRESS>/24
Client network: <IP_ADDRESS>/24
OpenVPN tunnel network: <IP_ADDRESS>/24

You need a client specific override with CN and set remote network there too

While it wasn't shown before, I do have that already on the server side, again configured as per the documentation

I realized my prior screenshot didn't show the CSO settings, so here's the actual settings

Please, select the proper tunnel in the field "Servers".
It solved for me.

Has anyone managed to solve it? I currently have the same problem. Proper tunnel in the field "Servers" i have selected.

Same here:
Warning openvpn_client1	ERROR: FreeBSD route add command failed: external program exited with error status: 1
Notice	openvpn_client1	/sbin/route add -net <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>
Notice	openvpn_client1	/sbin/route add -net <IP_ADDRESS> <IP_ADDRESS> <IP_ADDRESS>


permalinks do not work

I followed the instructions to set up a wordpress site with clevercloud. I'm able to access the home page and admin areas, but the permalinks for posts do not work, as you can see here: https://id5-test-wp.cleverapps.io/
Is there a setting that needs to be adjusted someplace to fix this? As mentioned, I followed the instructions exactly, without modifying anything else yet.
Thanks!

i don't think this is actually an issue with this repo...i'm having the same issue with bedrock directly, too.  closing this out.

Hi Scott, I haven't encountered this issue when rebuilding. I approved your merge request and I'm going to run some tests with both versions. Thanks for the feedback.


Multiple Saves (i.e. Browser, Sync, Periodic Manual Downloads, Discord [?] )

devs task. If it concerns the data team, please add more info on it so we can start working on it


use scalar projection for all-outbounds if we can

When an all-outbound path's value is needed for a page (either by being visible, or being used in the wait_for list of another column) we're modifying the main entity request to include the data for this path.
The current implementation doesn't differentiate between scalar and entity modes and will ask for the whole row instead of just the column that is being projected. For example, let's assume we're looking at table main where it has a foreign key relationship to table o1 through the following column mapping:
{
  "names": [["schema", "main_fk1"]],
  "foreign_key_columns": [{
    "column_name": "fk",
    "table_name": "main",
    "schema_name": "schema"
  }],
  "referenced_columns": [{
    "column_name": "RID",
    "table_name": "o1",
    "schema_name": "schema"
  }]
}

If we have a visible-column like the following
{ "source": [{"outbound": ["schema", "main_fk1"]}, "RID"], "entity": true}

It will result in the following request:
M:=s:main/F1:=left(fk)=(o1:RID)/RID;M:=array_d(M:*),F1:=array(F1:*)@sort(RID)

And with this when we're populating data for that visible-column we have the whole row data which allows us to,

Properly populate the "row name" of this foreign key relationship.
Access all the column data in the templating environment.
Honor the row_order for sorting.

But in case of scalar all-outbound, we might not need to fetch all the row values and we could potentially just fetch the column's value. To ensure that is the case, I added a canUseScalarProjection API to PseudoColumn which returns true if,

It's a scalar all-outbound path
The leaf column doesn't have  a customized  markdown_pattern in column-display annotation.
The leaf column doesn't have a column_order that is based on other columns.

With this we can ensure that just fetching the scalar column is enough.

The build in Github Actions started failing for reasons that are not directly related to this PR. The issue was related to jsdoc dependency. It seems like the latest 3.6.9 was causing these issues and fixing the version to 3.6.7 solved the issue.


feature request: Start/Stop charging switch

i've just found this integration, and am trying to get it working together with the EV Smart Charging integration (https://github.com/jonasbkarlsson/ev_smart_charging/).
is it possible to add a switch that Start/Stops the charging?

you mean this? "Socket 1 Operation Mode" where you can choose operative and inoperative?

yes, thats the one.
is it possible to convert it to a switch, because thats whats needed by the integration

yes, thats the one. is it possible to convert it to a switch, because thats whats needed by the integration

Is there an alternative method to circumvent the need for that switch? If I have to introduce a switch for that particular field, it would necessitate custom code modifications. This is because for the switch, we should have a boolean value of 0 or 1. If not, I would have to write custom code to manually convert the value from "operative" or "inoperative" to a boolean.

i think i can create a switch and use an automation to set the operative

@ronajon Did you manage to get it working? If so, how?

@ronajon Did you manage to get it working? If so, how?

unfortunately i did not


Update docs for extended delivery models

Motivation
Extends docs for https://github.com/kontent-ai/model-generator-net/issues/90
include modular content to structured models rendering
Notes
Merge after https://github.com/kontent-ai/model-generator-net/pull/165 is released

Codecov Report

Merging #375 (4144e50) into master (6cbac5d) will decrease coverage by 0.38%.
The diff coverage is n/a.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##           master     #375      +/-   ##
==========================================
- Coverage   88.87%   88.49%   -0.38%
==========================================
  Files         125      128       +3
  Lines        2679     2756      +77
  Branches      341      345       +4
==========================================
+ Hits         2381     2439      +58
- Misses        169      185      +16
- Partials      129      132       +3

see 13 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Out of date image in To access topics management settings:

The image in the section To access topics management settings: point 2 is out of date. Here is the updated image

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: e35fad7f-62a3-885b-c5fc-4be15ceeb7d7
Version Independent ID: e35fad7f-62a3-885b-c5fc-4be15ceeb7d7
Content: Change the name of the topic center in Microsoft Viva Topics
Content Source: Viva/topics/topic-experiences-administration.md
Service: viva
Sub-service: viva-topics
GitHub Login: @ruthholls
Microsoft Alias: ruthhollands

Image updated in https://github.com/MicrosoftDocs/Viva/commit/76a0876d0f7daea8d3b89e87b7727c3108687d38
Thanks!


clickhouse support

thanks you for your great efforts!
could you implements ClickHouse support for your data quality platform?
ClickHouse is blazing fast columnar database for analytics
look https://clickhouse.com/docs/en/ and https://github.com/ClickHouse/clickhouse-java for details

Support for ClickHouse is in the backlog, but so far, not a single potential user has asked for it.
If a paying user is willing to sponsor this feature, we will add it within four weeks.

We just added support for ClickHouse in the development version of DQOps. It will be released in the upcoming version.
If you want to test it, you can try it by compiling the code from the "develop" branch.


Custom element on last (or second to last) line breaks the parser

See https://ellie-app.com/3XXCryvVrzba1
It works if you add a blank line at the end of the markup. So the workaround is easy in the example, but gets tricky when the content is in separate file. Many text editors have a feature of removing trailing empty lines. That's how I've run into it.

I believe this is fixed in v2 because blocks are no longer closed by 2 newlines, but by a change in indentation.


Organize the Project

Organize the project into separate source files.

git fetch origin
git checkout 3-organize-the-project


Increment version for digitaltwins releases

Increment package version after release of Azure.DigitalTwins.Core

@weshaggard, @sima-zhu:  This failure doesn't look right to me.   Would you mind offering your thoughts?

That does not look correct to me either @sima-zhu can you please investigate.

@jsquire @weshaggard The credscan yaml file is out of date to the feature branch. We can either pull latest or check-pick my commit, or trigger aggregate-report for the secret scanning. If no secret, we can safely bypass the error

Secret scanning triggered for the feature branch: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1465188&view=logs&j=3b141548-98d7-5be1-7ef8-eeb08ca02972&t=d503b138-f32a-5a02-e24d-80f2b5d88749

https://dev.azure.com/azure-sdk/internal/_build/results?buildId=1465188&view=logs&j=3b141548-98d7-5be1-7ef8-eeb08ca02972&t=3b141548-98d7-5be1-7ef8-eeb08ca02972


Youtube streams do not work.

Youtube / soundcloud streams do not load, I add them but they do not show up on the queue. The local and the radio function works fine but the youtube bit does not.

They load just fine for me. What URI are you putting into the streams box? For youtube I don't think you can just use the normal HTTP URI like you would in a web browser. For example, for youtube video https://www.youtube.com/watch?v=aAst-vyasiM, you would enter yt:aAst-vyasiM. Sometimes it takes a little while to load and start playing.
soundcloud should just work with the normal webpage URL.

@kingosticks I was using the normal url but i have tried yt:aAst-vyasiM and aAst-vyasiM but they do not load / play.

 If you look in the Mopidy log you may find error messages
explaining why it's not playing for you. It certainly worked for me.
On 3 May 2015 21:13, "chipyzs"<EMAIL_ADDRESS>wrote:

@kingosticks https://github.com/kingosticks I was using the normal url
but i have tried yt:aAst-vyasiM and aAst-vyasiM but they do not load / play.
—
Reply to this email directly or view it on GitHub
https://github.com/woutervanwijk/Mopidy-MusicBox-Webclient/issues/102#issuecomment-98536402
.

I'm also running into this issue with youtube & soundcloud. I've monitored the Mopidy logs but nothing relevant came up. I've been trouble shooting for an hour now so I'll update this if I find anything out.
I'm running Ubuntu 12.04 LTS

@javaj0hn, Just to be clear, do youtube and soundcloud streams work from the search results?

@kingosticks Just saw this was a feature. No they do not load any search results.

In which case this doesn't sound like a webclient issue.
You should ensure the extensions are definitely loaded and enabled, this should be clear in the log file. mopidy deps and mopidy config might also help you determine this.

My deps & config are as follows - http://pastebin.com/cTggGkir
I feel that I am mis-configuring something with the websockets.

There is nothing to configure for websockets for normal users. And normally you would not set musicbox = true either.
Neither Mopidy-Soundcloud or Mopidy-Youtube exist in your deps or your config which shows that Mopidy is not finding them on your system. Are you sure they are installed? Have a look at https://docs.mopidy.com/en/latest/installation/debian/#missing-extensions and  https://docs.mopidy.com/en/latest/debian/#differences-when-running-as-a-system-service

Did you get this working? I don't think there is a webclient issue here.

Hey apologies, just saw this updated. Yes I got it working, I was missing the youtube and spotify deps. All solved.

That's great and no worries.


Regenerating PACs

After adding rustfmt in https://github.com/esp-rs/esp-pacs/pull/257 , a lot of changes in (not only) docs appears after generating PACs.
Feel free to dismiss it in case of mistake

Oh, also there's a new svd2rust patch (0.33.4), which also seems to cause that amount of changes. So it's a combination of rustfmt and svd2rust new patch version.
On previous 0.33.3, rustfmt doesn't cause this amount of changes (something around 90 "only")


mannwhitneyu acceptes nans

Not sure if this is a feature request or a bug.
mannwhitneyu should not accept np.nan, currently, it accepts it and gives bad p_values.
import scipy.stats as stats
import numpy as np

a = [1,2,np.nan,np.nan,np.nan,np.nan]
b = [1,2,np.nan,np.nan,np.nan,np.nan]
stats.mannwhitneyu(a,b).pvalue
#  0.0281021903842455

This is pretty dangerous, especially when working with dataframes. I assume adding nan_policy to mannwhitneyu will be good (have not seen a ticket for that), but until it is added, maybe just returning an error if one of the vectors has np.nan (if None exists an error is already returned)
scipy.version
'1.6.3'

Sorry, it already exisits here:
https://github.com/scipy/scipy/blob/44e31b1eb202ef91caad059952ba268d2610a60b/scipy/stats/_mannwhitneyu.py#L132

Yes, i rewrote MWU for 1.7.0. it can now do the exact test for samples of moderate size, and it's vectorized / supports an axis argument. My plan is to add nan_policy with a decorator (gh-13312).


Update logger dependency

logger_flutter 0.7.1 depends on logger ^0.7.0 and no versions of logger_flutter match >0.7.1 <0.8.0, logger_flutter ^0.7.1 requires logger ^0.7.0.

https://github.com/leisim/logger_flutter/pull/7

You can always do something like:

logger_flutter:
git:
url: https://github.com/JEuler/logger_flutter.git

But preferable if you fork it into your own acc, because I can remove it sometime somehow. :)

Could @leisim please merge the PR?

+1 for this, because I'm using flutter web and I need the logconsole to work with the current patched logger 0.9.1 ;) At the moment I'm using the workaournd by Jeuler in pubspec.yaml but I think its better to release a new version with the dependency fixed

Yeah, it would be really nice if you release new version, @leisim :) https://github.com/leisim/logger_flutter/pull/8 I create new PR for this, I'm sure you know how to publish update for package :D
But anyway, this will make it easier.

Well I tried to use newest logger together with current logconsole and I got errors, because a Callback method is not supported in the current logger version.  I dont know how to fix this myself but I think this needs some compatibility work to work with the latest logger version? Can anybody verify this?

Yeah, it is not working for the last version. I've checked. Hm.

So we need to use LogOutput. I will have a look later on weekend days. :)

Using the workaround provided, but a gentle +1 that an update would be appreciated. :)

still has dependency error on the official package

Any update on this?

this is working for me :
logger: 0.7.0+1
logger_flutter: ^0.7.1

Hey @leisim any update on this? The issue persists! :)

https://github.com/leisim/logger_flutter/pull/9/files - maybe you can use this PR.
logger_flutter:
git:
	url: https://github.com/ewertonrp/logger_flutter

Like this
All credits to @ewertonrp

For me this worked:
logger_flutter:
    git: https://github.com/ewertonrp/logger_flutter.git

Seems to break again with the nullsafe version of logger:
logger: ^1.0.0-nullsafety.0 #^0.9.4
  logger_flutter: #^0.7.1
    git: https://github.com/ewertonrp/logger_flutter.git

Because every version of logger_flutter from git depends on logger >=0.7.0 <1.0.0 and logging_test depends on logger ^1.0.0-nullsafety.0, logger_flutter from git is forbidden.
So, because logging_test depends on logger_flutter from git, version solving failed.
pub get failed (1; So, because logging_test depends on logger_flutter from git, version solving failed.)


Apply transformation when Patching from Resource

I'm trying to create a k8s secret patching from another secret.
I'm trying to achieve this by applying the following object inside of a composition:
- name: argocd-creds-secret
      base:
        apiVersion: kubernetes.crossplane.io/v1alpha1
        kind: Object
        metadata:
          name: cluster-bearer-token
        spec:
          references:
          - patchesFrom:
              apiVersion: v1
              kind: Secret
              name: connection-string
              namespace: default
              fieldPath: data.kubeconfig
            toFieldPath: data.bearer-token
            transforms:
              - type: string
                string:
                  type: Regexp
                  regexp:
                    match: '(?<=token:).*$'
          forProvider:
            manifest:
              apiVersion: v1
              kind: Secret
              metadata:
                namespace: default
              data:
                sample-key: #to be patched
          providerConfigRef:
            name: local-kubernetes-provider

It looks like it completely ignores the transformation.
Is the transformation taken into account at the current implementation=

@fabioaraujopt yes, transforms was not implemented intentionally at the beginning when patchesFrom was introduced as the more we add such at provider level, the more we see the similarity for such feature in Composition at core level. Will see if there are more such requirements from community. @turkenh ?

Thank you for explanation @morningspace. Do you suggest any alternative to this? We are increase a lot the complexity of several compositions because of this issue.

We also have the requirement for transformations, also to e.g. combine multiple different fields into a single string in the target object

We also need this, for example to add a prefix or suffix when patching a value

This would be a nice addition to the provider :)

Would love to see that as well.

Patch and Transforms can be quite complicated and are provide limitations in some cases (like lack of certain transform features). I would suggest using something more generic and more "common" in the Go and K8s ecosystem, like Go templates.

+1, patching from a secret is impossible right now because it would need a base64 decode transformation. Writing a XRD, composition and claim for it is cumbersome and needs a lot of glue yaml (patch to status field etc.)


Problem when snoop to boudingbox element

I input a current view to get the result.
CommandData.View

I don't konw weather i can get all of the views in doc.

no need with boudingbox, we can input null is default to getboudingbox,
Example :
element.GetBoudingbox(null);

amazing,i don't know about this.

whether we can set defaul is null for user ? we still use dropdown if user cannot get boudingbox with value null.

we can get all view in doc, use some like that
private List<View> view;
        public List<View> AllViews
        {
            get
            {
                if (view == null)
                    view = new List<Element>(new FilteredElementCollector(doc)
                            .OfClass(typeof(View)).WhereElementIsNotElementType().ToList().OrderBy(v => v.Name)).Distinct()
                        .ToList().Cast<View>().ToList().Where(x => !x.Name.Contains($"<Revision Schedule>")).Where(x => !x.IsTemplate).ToList();
                return view;
            }
            set
            {
                view = value;
            }
        }

Yes,we can set default value as null, let me take a try.


Fix type-only imports in interface 'extends' and import=/export=

Fixes a bug I discovered while working on this where errors for using a transitively type-only alias didn’t appear in editors depending on what files are open (because alias resolution happened in a different order than in tsc)—tested in the added unit test.
Fixes #36478
Fixes #36466

Disallows import aliases referencing type-only names
Fixes ImportEqualsDeclarations and ExportAssignments referencing type-only names (adds appropriate error for usage of alias in JS-emitting positions)

I just commented on #36478 as I don't think this fixes the issue when the type is another level deep because it appears I can do this:
import type * as monaco from "monaco-editor";

interface MyViewZone extends monaco.IRange {
  marginDomNode: HTMLElement;
}

but not this:
import type * as monaco from "monaco-editor";

interface MyViewZone extends monaco.editor.IViewZone {
  marginDomNode: HTMLElement;
}

Ah, though as a workaround I can do:
import type * as monaco from "monaco-editor";
type IViewZone = monaco.editor.IViewZone;

interface MyViewZone extends IViewZone {
  marginDomNode: HTMLElement;
}


correct pyenv path

rename profile into zprofile so that it is sourced by zsh
this corrects https://github.com/lewagon/dotfiles/pull/90 which introduces a .profile conf for the first part of the pyenvsetup. The .profile is not sourced whereas the .zprofile is sourced by zsh before the .zshrc is sourced

Using a .zprofile seems to work on Parallels / Windows / Ubuntu (I had a warning proposing to add pyenv init - to the .zshrc until the VM was restarted).
For Mac the pyenv install steps for zsh differ depending on how pyenv was installed (git vs brew). On my machine I have both at the moment 🤦♂ and no issues. Maybe forcing PYENV_ROOT for brew installs is an issue.

Thanks Damien 🙏 🔥


boost::diagnostic_information() works in no rtti mode.

boost::exception processes BOOST_NO_TYPEID and BOOST_NO_RTTI inside more flexible.

I see a lot of C++11 test failures. Is it normal?

Yes, the travis and appveyor need some rework.

All good, in next, should be there for 1.72!
Thanks!

In master


Include only selected pojo from a package , while validating

Hi team,
Can you please help , how to include selected pojo from a package  and use directly with the validate method. how to provide directly the class in the param.

If you want a single Pojo to be validated, you can do the following:
  @Test
  public void someTestMethod() {
    Validator validator = ValidatorBuilder.create()
                            // Add Rules see com.openpojo.validation.rule.impl for more ...
                            .with(new SomeRule())
                            // Add Testers see com.openpojo.validation.test.impl for more ...
                            .with(new someTester())
                            .build();

    PojoClass classToValidate = PojoClassFactory.getPojoClass(com.package.ClassToValidate);
    validator.validate(classToValidate);
  }```

Another way is to use PojoClassFactory.getPojoClasses("package name", "filter to select just what you need");


return data

I'm fairly new to async js.  I'm hoping you can help me.
My forEachAsync calls a function that also contains a forEachAsync.  How do I return data from child loop to parent loop?
element contains a child array of photos that I have to convert to a base64 and return back as a array of photos with their new dataUrl. I need to update the photos in all of the docs.
hope this makes sense.  I'd appreciate any help.
Thanks,
Donnie
forEachAsync(docs, function (next, element, index, array) {
  convertPics(element,next);
}).then(function () {

});

function convertPics(element,cb){
    var picJson = JSON.parse(element.photos)
    var basedir = process.env.CLOUD_DIR+'/';
    var arr = [];

    forEachAsync(picJson, function (next, p, index, array) {

        if (fs.existsSync(basedir + 'thumb_' + p.fileName)) {

            p.dataUrl = fs.readFileSync(basedir + 'thumb_' + p.fileName).toString('base64')

            arr.push(p);

        }
    }).then(function () {
        ???? how do I return the array so I can use it in the then of the parent call?
       cb() ???
    });

}

Delete the word 'async' from your mind and replace it with 'event'.
JavaScript doesn't operate from one line to the next. It operates on events.  A mouse click is an event. A file read is an event. Every action that requires waiting is an event.
I think this is what you're trying to do, but I'm not sure:
var basedir = process.env.CLOUD_DIR+'/';
var arr = [];

function doStuffWithDocs() {
  return forEachAsync(docs, function (next, item) {
    convertPics(item, next);
  });
}

function convertPics(element,cb) {
  var picJson = JSON.parse(element.photos); // is this not already parsed?

  forEachAsync(picJson, function (next, p) {
    fs.exists(basedir + 'thumb_' + p.fileName, function (exists) {
      if (!exists) {
        next();
        return;
      }

      fs.readFile(basedir + 'thumb_' + p.fileName, function (err, buffer) {
        if (err) {
          console.error('error reading thumb file:');
          console.error(err.stack);
          return;
        }

        p.dataUrl = buffer.toString('base64');
        arr.push(p);
        next();
      });
    });
  });
}

doStuffWithDocs(docs).then(function () {
    doStuffWithAllPics(arr);
})

Avoid anything *Sync at all costs because it's just gonna cause more confusion.
Events. Events. Events.

Updated a few things.

Wow! Thank you. This is exactly what I was trying to do.   Love the sandwich metaphor. :)   Clearly I have a lot to learn. Recommend any good books,videos, etc. for getting better at this?

Honestly, it's really tough to sort the good from the bad.
My favorites are the Crockford vids. You won't understand them in the first go. I watched Function the Ultimate probably 3 or 4 times before I understood it:
https://www.youtube.com/playlist?list=PLlRSdZcxPj9_GK2i1KVvnZD9fVToABz3V
The learn JavaScript in 30 minutes series I've only watched a few of, but it was really good:
https://www.youtube.com/playlist?list=PL5C542E0286878FCB

You might also benefit from learning a little about ruby's EventMachine. This framework was contemporary to (and perhaps somewhat an inspiration for) node.js
https://vimeo.com/10849958

excellent! I'll check them out.  Thank you!

@coolaj86

Honestly, it's really tough to sort the good from the bad. So many different opinions and so few people teach JS as Event-Oriented. They try to cram it in with Object-Oriented, but it's just not the same.
My favorites are the Crockford vids. You won't understand them in the first go. I watched Function the Ultimate probably 3 or 4 times before I understood it:

Firstly 👍 👍 👍 for recommending Crockford's lectures. They are by far the better resource for learning JavaScript. Even will all the new features being added at an ever faster cadence, his advice (pragmatism founded in experience and theory), practices (emphasis on correctness and readability), and teaching style (rigor, humor, and historical perspective) remain as relevant and valuable as ever.
Now regarding the whole issue of people trying to pretend that JavaScript is

Object-Oriented

I disagree slightly. It is a very real phenomenon and a pretty massive problem, but I would frame it differently. JavaScript is Object-Oriented, but most people think that this implies a Java-like language, something very different.
Regardless, my feeling is that the biggest problem here is overuse of classes, especially when such use involves conflating the notion of a JS value with that of a type. instanceof is a terrible thing.

Events. Events. Events.
You generally don't "return" an object to a parent function (except for basic math functions), you call the next function you want to use (as if it were an event completion) with the result of whatever event you're working on.

This is where I break with you. Although in principal what you say is entirely correct, there are two significant problems with this approach.
Firstly, it assumes that while one may (must?) learn JavaScript as a largely event driven language and this is indeed fine, continuing to program in a callback based style makes code illegible and we can and should abstract from it where possible by leveraging both promise oriented libraries, like yours, and also by taking advantage of syntactic features like the await operator. If we have a suitable abstraction, we should only think in terms of events for things that are logically events and not, for example, when reading a file from beginning to end.
Secondly, you seem to be intimating (I may be misreading) that wanting to abstract over events and compose them and the values that they in turn bring into being in a value like manner is essentially applying a narrow, Object-Oriented mindset to JavaScript. I would argue it is if anything applying a functional mindset, but why bother with paradematic comparisons. Let's keep using promises and the new ES primitives to make our code as readable as possible. Btw, this library works really nicely with async and await.


Review ACS Reports

Review ACS reports and come up with any necessary update plans.

[ ] backup images

Would require the base image for the backup container to be upgraded.
Currently using a centos/postgresql image as the base image.  The centos base images are now deprecated.
We should consider moving to the official Postgres images for the base.  There are rhel based images available, but they require an account or token to get, which is fine if you're building the image on the platform.

[ ] database images

Review and update (rebuild) from the latest base images.
Reviewed Orgbook environment.

[ ] Orgbook api images

Update (rebuild) from the latest base image.

[x] Orgbook frontend images

They are ok..

[ ] agent images

Test out an image based on the Docker.indy build from this aca-py PR, https://github.com/hyperledger/aries-cloudagent-python/pull/1888.
To get a more up-to-date base image build the image using python 3.9.  The aca-py tests all seem to pass when run on python 3.9 images; https://github.com/WadeBarnes/aries-cloudagent-python/actions/runs/3528226011/jobs/5918100044

[ ] search-engine images

Try upgrading Solr to v9.1.
This involves updating the https://github.com/bcgov/openshift-solr project to use v9.1 as the base image, and then rebuilding the search-engine images using the new base image.  Thorough testing needs to be done with the existing indexes, and Solr recommends rebuilding search indexes when upgrading between major releases.  That would require using the offline-indexing pods and swapping out the indexes.

Will need to break down the outcomes in multiple stories/spikes

Closing this in favor of https://github.com/bcgov/DITP-DevOps/issues/158.  This issue has become a bit stale, but is referenced in the new ticket for reference.


Custom Shader batch draw needs refine

RT
see also: https://github.com/c4games/cocos-re/issues/234

This issue is not specific to Spine nodes, here's how sprites look in Material System test (cpp-tests).


MySQL vs SQLite

Figure out which is easiest to manage..

Decided on MySQL because there's a lot of help on the internet for it. Also, I'm already using tutorials that utilize it.


fix: use the main prop for the application entry point.

npm start will now look at the main property for the entry point.  Port 8080 is now set as the default port instead of 3000.   This is a fix related to https://github.com/bucharest-gold/centos7-s2i-nodejs/issues/33#issuecomment-382587104.
fixes #98

Coverage remained the same at 93.75% when pulling 34b8c91281e24d11b36e06b773c4ffaa58206fba on lholmquist:98-change-start-script-and-port into 3550fa41300ba14712f2f16e215a4fc4325e370d on bucharest-gold:master.


Added default values for smart content view vars

Tasks:

[ ] gather feedback for my changes
[x] added changelog line

Informations:

Q
A

Tests pass?
yes

Fixed tickets
none

BC Breaks
none

Doc
https://github.com/sulu-cmf/docs/commit/040257e43fc7d50d5c8b8f571278caffb811974c

@dantleech i will look at the tests (=

@dantleech fixed testcases (they should work)

Rebase and merge is fine.


update to refer to newer version of beakerx widgets

this has a fix to prevent the widget from jumping down when window is narrow, plus a big performance fix.  here are comparison screenshots showing the bug visible on the current site, and then the fixed version:

@SylvainCorlay @jasongrout

Thanks!


Add comment about open-browser flag

I couldn't find any reference to this flag. I see now there's another doc that has it, but that didn't come up when I was googling. Having it on this doc too will help I think.

Thanks!


Create a custom Input component with value formatter

I am trying to implement a custom CustomInput for a form guiding me in the shadcn input component standard. Basically I want that when a user types, a formatting function is applied to the text, but when the form is submitted, the CustomInput returns the value without the formatting function applied.
Right now I have something like this, which is formatting the value on change, but is returning also the value formatted.
const CustomInput = React.forwardRef<HTMLInputElement, RutInputProps>(
  ({ className, ...props }, ref) => {
    return (
      <input
        ref={ref}
        {...props}
        onChange={(e) => {
          e.target.value = formatValue(e.target.value)
          props.onChange?.(e)
        }}
        value={props.value as string}
        className={cn(
          'flex h-10 w-full rounded-md border border-input bg-background px-3 py-2 text-sm ring-offset-background file:border-0 file:bg-transparent file:text-sm file:font-medium placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:cursor-not-allowed disabled:opacity-50',
          className,
        )}
      />
    )
  },
)

This issue has been automatically closed because it received no activity for a while. If you think it was closed by accident, please leave a comment. Thank you.

i also need this


Uniprot improvements

switch from stream to search API (stream only supports small datasets https://www.uniprot.org/help/api_queries)
add TSV output (possible columns are derived via a call to the uniprot API :))
small wording changes

Linting error will be fixed by https://github.com/galaxyproject/galaxy/pull/19300


added Query Interface and some doc fixes

Keep getting duplicate docstring warning: https://github.com/QuantumSavory/QuantumSavory.jl/actions/runs/7744890873/job/21119667800?pr=92#step:5:1383
in the log. It triggers the duplicate docstring warning due to same object being  put inside @docs template more than once even if the signatures are clearly different


feature/SMHE-1962-postprocess-response-of-set-and-get-configuration-object-requests

After Getting and Setting Configuration Object hls3active, hls4active and hls5active attributes of the dlms device should be updated in the dlms adapter database.

⏳ https://jenkins.fdp.osgp.cloud/view/SMHE-cucumber/job/gxf-smhe-cucumber-build/198/


Removing dashes

The task is to resolve issue #237  by replacing all dashes with underquotes.

This was already fixed in #245

Okay i understand but would like to take look as there are still dashes issue in .well folder


Quickstart with devcontainers in VSCode and fix some CMakeLists.txt files

add support for devcontainer with VSCode and update Readme.md to reflect that
correct add_executable in CMakeLists.txt for fltk and gtkmm projects
correct handling of -ftime-trace option in main CMakeLists.txt

Fix #161

OP here, I won't be looking any further into this as its been a while since I've used vscode or any cpp related stuff and don't have the bandwidth to put myself back into those ecosystems
Feel free to rebase this and apply changes as necessary

OP here, I won't be looking any further into this as its been a while since I've used vscode or any cpp related stuff and don't have the bandwidth to put myself back into those ecosystems
Feel free to rebase this and apply changes as necessary

That's OK @midnightexigent thank you for letting us know!

Superseded by #175


Empty object with animation component crashes when the animation is played

Bug Environment:
Engine
Version/Release:
Currently on develop
Bug Description
When creating many GameObjects in hierarchy, and adding an animation component, it crashes when the animation is played.
Expected Behaviour:
Nothing happens as you press the button.
Actual Behaviour:
Game crashes.
Steps to Reproduce:
Create a GameObject from hierarchy. Give it a Animation Component. Play the animation.

void InspectorPanel::DrawAnimationComponent(AnimationComponent* component) {

	ImGui::SeparatorText("Animation");
	ImGui::Text("HELLO");

	static bool play = false;

	if(ImGui::Button("Play"))
	{
		if (component->GetAnimation() == nullptr) //Always check if null before operating things!!!!
			return;
		component->OnStart();

		play = true;
	}

	if(play)
		component->OnUpdate();

}


Clicking Docs shouldn't change you to TypeScript

Say I'm in the Dart guide: https://angular.io/docs/dart/latest/guide/. When I click DOCS at the top of the page, it takes me to the TypeScript doc homepage (https://angular.io/docs/ts/latest/).
I'd expect to be taken to the Dart doc homepage: https://angular.io/dart (or, if need be, https://angular.io/docs/dart/latest/).
Without that, there's no way to get back to the Dart doc homepage.

In general, I'd like to stay on the Dart docs unless I explicitly go to the TS docs. That means once I've gone to docs under docs/dart, I'd expect the drop-down list to default to Dart until I explicitly go to a docs/ts or docs/js page.

Done in #1144


PatchGenerator does not support certain image sizes

Describe the bug
When attempting to run inference with a ResNet50 patch-wise classifier using FastPathology, it was observed that image size 299 x 299 was not supported.
According to the source code, FAST does not support image patches that are not divisible by 2 (see here).
System:

OS and version: [e.g. Windows 10/Ubuntu 18.04] Ubuntu 20.04
GPU and OpenCL vendor (NVIDIA, AMD, Intel) NVIDIA

Expected behavior
The PatchGenerator PO should support "any" input shape
cc: @markusdrange

This is not really a bug. TIFF requires the tile size to be a multiplum of 16. TIFF is used in the PatchStitcher when creating a segmentation ImagePyramid on the fly. The TIFF tile size could be different to the patch generator size, but this would lead to very inefficient stitching.

Hmm, I understand. Perhaps we should add a disclaimer regarding this, either in FAST or in FP to inform users about this, as I believe there will be people that may run into the same issue. 299x299 is common for ResNets and Inception backbones.


[Merge rocksdb in datanode] Rebase HDDS-3630 on master.

Rebase HDDS-3630 on master so we could catch up.

cc @ChenSammi

All CI green, so I got to do a force-push to branch HDDS-3630.

Pushed, close this.


Allow launching a Godot version without creating or opening a project

Is your feature request related to a problem? Please describe.
In the Godot Versions panel, I want to launch a Godot version.
Describe the solution you'd like
Just launching a Godot version.
Describe alternatives you've considered
Copying the installation path and opening directly from there.

This will be something I will add to the next point release, and 0.3.x release of Godot Manager.


May you add sprinkler/Valve support?

I have a irrigator based on ewelink module.the model is PSF-B04-GL.It has 2 channels switch.It works fine as 2 channel switches with this plugin.But I want to work as valve/sprinkler to get the timer function.Would you tell me how to make the changes or would you release an update to support the sprinkler/valve/irrigation type?
Thank you so much for your contribution.

Hi,
Ooh new devices is my favourite part of maintaining this package.
I would need some detailed information from you about the device, would you be happy to send me an email?
<EMAIL_ADDRESS>

Hi,
Ooh new devices is my favourite part of maintaining this package.
I would need some detailed information from you about the device, would you be happy to send me an email?
<EMAIL_ADDRESS>
Thank you so much.I will send you more detail about the new device type later.


Question: cluster update with linclust algorithm.

Hi, Thanks always for the nice tool!
I have clustered a large sequence set with mmseqs linclust , and now I want to add some sequences and update the clustering result with mmseqs clusterupdate.
But it seemed that mmseqs clusterupdate updates clustering results with the mmseqs cluster algorithm but not mmseqs linclust .
I am guessing this because all the parameters of mmseqs clusterupdate match with mmseqs cluster, but not mmseqs linclust .
Is it correct?
Is it possible to update clustering with mmseqs linclust  algorithm?
Thanks in advance.

Uh, I have the same question.


Unable to open the app on a Mac

I've installed Homebrew and downloaded the MacApp but I'm getting the following message.

Expected Behavior
Able to open the MacApp link
Specifications
Platform: Mac
OS: MacOs High Sierra 10.13.6
App version: Miscord 4.7.1

It's a known issue, but I cannot reproduce it.

Basically, I'm not a certified Apple Developer, so people would need to disable System Integrity Protection...

You could probably fundraise for this one.

It could be a Patreon goal, I guess.

$99 in a year should be doable, although I'd probably suggest a specific fundraiser, not necessarily Patreon

You should probably take down the link to download the app for mac on the miscord website. I just downloaded it and ran into the above issue.

damn, you're right


Separate initilizers from inputs in target opset 9+

No, we want the final container.initializers separate from container.input. It is fine to put initializers into extra_inputs for optimizer.


cmd/go: build fails if filename is provided

What version of Go are you using (go version)?
go1.11beta3
Does this issue reproduce with the latest release?
Doesn't repro on go1.10 or go1.11beta2
What operating system and processor architecture are you using (go env)?
GOARCH="amd64"
GOOS="darwin"
What did you do?
Made single main.go hello world app with one line go.mod file
Ran go build -o bin/mod-test main.go
What did you expect to see?
Successful build
What did you see instead?
Build error:
build main: cannot find module for path main
If I run the build without the filename
go build -o bin/mod-test it works fine
This is a change from previous versions of Go.

The go build command's arguments are interpreted only as packages, not files. You are asking to build the package "main.go", and there is no such package. See the output of "go help build", which begins:
usage: go build [-o output] [-i] [build flags] [packages]

Build compiles the packages named by the import paths,
along with their dependencies, but it does not install the results.

Our Makefiles have been building using the above syntax for years. I'll fix them up to be more correct.
This change in behavior, even though more correct, might want to be called out in the release notes.  There may be other teams using go build incorrectly as well.

This also doesn't when using a package name
go build main
go: import "main": cannot find module providing package main

CC @bcmills
This works differently in module mode vs GOPATH mode
root@3a67ccb3e0d7:/go/src/mod-test# ls
main.go
root@3a67ccb3e0d7:/go/src/mod-test# cat main.go
package main

import "fmt"

func main() {
	fmt.Println("Hello World.")
}
root@3a67ccb3e0d7:/go/src/mod-test# go build main.go       ## this works
root@3a67ccb3e0d7:/go/src/mod-test# go build main            ## fails
can't load package: package main: cannot find package "main" in any of:
	/usr/local/go/src/main (from $GOROOT)
	/go/src/main (from $GOPATH)

Not sure quite what's happening here, but it sounds like there has been a change in behavior. Re-opening.

Similarly executing go run main.go in a module directory results in errors:
$ go run main.go
build main: cannot find module for path main
$ go run main
go: import "main": cannot find module providing package main

However go run . works as expected.

This is a departure from previous behavior if we cannot specify a list of .go files to "go build".
The output of "go help build" continues:
If the arguments to build are a list of .go files, build treats
them as a list of source files specifying a single package.

When compiling a single main package, build writes
the resulting executable to an output file named after
the first source file ('go build ed.go rx.go' writes 'ed' or 'ed.exe') ...

This has the same underlying cause as https://github.com/golang/go/issues/26722; so closing as a duplicate.
I've confirmed that the fix in https://go-review.googlesource.com/c/go/+/127796 for that issue fixes this case:
cd `mktemp -d`
export GOPATH=$(mktemp -d)
mkdir hello
cd hello
go mod init example.com/hello
cat <<EOD > main.go

package main

import "fmt"

func main() {
        fmt.Println("==========")
}
EOD
go build main.go
./main

gives:
==========

whereas on tip (5a720d229d5ebc2f8b599aa54a3d807a1d54365f) this fails.


Remove integrations and MIME_TYPE guessing

Resolves #263.
Ping @mo8it.

So much code removed. :3


cannot play, display or download a music generating prompt

Bing gpt says "listen my music" but no way to hear it.

Yes, me too


systemd-stage-1/plymouth: Password prompt takes a long time to appear

Describe the bug
With both systemd-initrd and plymouth enabled, the password prompt takes a significant amount of time to appear; about 8 seconds. The logs indicate that plymouth is ready and idle during that whole time, and there's no messages indicating an event to trigger the prompt to appear when it does. If the splash kernel parameter is removed, then the console-based password prompt appears immediately as expected.
Steps To Reproduce

boot.initrd.systemd.enable = true;
boot.plymouth.enable = true;
Use an encrypted root file system.

Expected behavior
The plymouth password prompt should appear extremely quickly after initrd starts.
/cc @littlejohn24

I've noticed that the delay doesn't occur with plymouth enabled when using systemd boot from lanzaboote. With no lanzaboote (and no secure boot) I was getting this on every boot.

Finally found something about this. If I enable plymouth debug logging, I see this:
00:00:00.659 ../src/libply-splash-core/ply-device-manager.c:498:create_devi: found device /sys/devices/platform/simple-framebuffer.0/drm/card0
00:00:00.659 ../src/libply-splash-core/ply-device-manager.c:505:create_devi: device is initialized
00:00:00.660 ../src/libply-splash-core/ply-device-manager.c:509:create_devi: found node /dev/dri/card0
00:00:00.660 ../src/libply-splash-core/ply-device-manager.c:396:create_devi: device subsystem is drm
00:00:00.661 ../src/libply-splash-core/ply-device-manager.c:400:create_devi: ignoring since we only handle SimpleDRM devices after timeout

And then 8 seconds later,
00:00:08.663 ../src/libply-splash-core/ply-device-manager.c:1237:create_dev: Timeout elapsed, looking for devices from udev

and the splash shows up.
So whatever this "SimpleDRM" thing is, plymouth doesn't like it.

Yea, plymouth deliberately refuses to use  simpledrm devices. You can use plymouth.use-simpledrm to override this, but I think this has the drawback that once amdgpu (or your equivalent) loads, the splash dies and never recovers. I'm not sure that's how that's working though.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/get-plymouth-to-show-password-prompt-for-encrypted-root/24916/9

I've noticed that the delay doesn't occur with plymouth enabled when using systemd boot from lanzaboote. With no lanzaboote (and no secure boot) I was getting this on every boot.

Is it possible that there's something wrong with our systemd-boot?

I've noticed that the delay doesn't occur with plymouth enabled when using systemd boot from lanzaboote. With no lanzaboote (and no secure boot) I was getting this on every boot.

Using lanzaboote and it is still delaying.

I've also noticed this. it isn't a big deal. just annoying

Yea, like I said, Plymouth just doesn't like SimpleDRM graphics, so it sits around for 8 seconds before giving up and using it anyway. The problem "goes away" if you include your graphics drivers in initrd (though this causes an unsightly graphical flicker during stage 1 that likely also takes some time), or you can force it to use SimpleDRM with plymouth.use-simpledrm on the cmdline.

https://gitlab.freedesktop.org/plymouth/plymouth/-/merge_requests/323

@misuzu That's interesting; that change seems to resolve an issue where simpledrm takes too long to be initialized, which is not at all what we're seeing here. In NixOS, simpledrm isn't used at all at first unless you pass plymouth.use-simpledrm on the cmdline, which is why it waits 8 seconds to fall back to it. The issue the describe in that merge request says that even simpledrm was timing out and it was falling all the way back to text mode.

This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/oem-logo-blinks-plymouth-not-shown-correctly/55490/2


Add Github actions to automate the release process on npm

Problem
Currently the release on npm is done manually
Solution
Add Github actions to automate the release process on npm
Alternatives
No response
Additional context
https://www.npmjs.com/package/ansible-language-server

I could look into this.

The token to use in CI is ${{ secrets.NPM_TOKEN }}. It is only available in the publishing environment. The other token would be ${{ secrets.GITHUB_TOKEN }}, to be used for GitHub Packages.

Now that we have access to the namespace, we also need to decide on the naming: @ansible/language-server vs @ansible/ansible-language-server.

Closing because we do have a process for it.


ScanOperationConfig.ScanFilter does not work as expected with nested properties

Description
When scanning on a nested property using the ScanFilter no results will be returned. Using the same expression in FilterExpression does result in results. So there seems to be a difference in using the ScanFilter and FilterExpression properties of the ScanOperationConfig. The ScanFilter does not seem to handle nested properties as expected.
Could be related to https://github.com/aws/aws-sdk-net/issues/1805 . But FilterExpression actually gives the correct result without splitting the nested property name as explained in #1805.
Reproduction Steps
Example data structure:
{
  "id": {
    "S": "23a19237-960b-476a-86c0-42eafa2bb1e8"
  },
  "info": {
    "M": {
      "lastMeasurementTimestamp": {
        "S": "2022-04-07T04:34:02+00:00"
      }
    }
  }
}

With ScanFilter (which does NOT give results):
var filter = new ScanFilter();
filter.AddCondition("info.lastMeasurementTimestamp", ScanOperator.LessThan, date);

var search = table.Scan(new ScanOperationConfig
{
	Filter = filter,
});

With FilterExpression (gives results):
var search = table.Scan(new ScanOperationConfig
{
	FilterExpression = new()
	{
		ExpressionStatement = "info.lastMeasurementTimestamp < :date",
		ExpressionAttributeValues = new() { {":date", date} }
	},
});

Logs

Environment

SDK Version: -
Package Version: <IP_ADDRESS>
OS Info: AmazonLinux
Build Environment: Visual Studio
Targeted .NET Platform: net5

Resolution

This is a :bug: bug-report

Hi @denxorz,
Good morning.
Thanks for reporting the issue. Going by the https://github.com/aws/aws-sdk-net/issues/1805#issuecomment-788125181, this behavior is by design. AWS SDK for .NET has the same behavior as AWS Console. Also refer https://github.com/aws/aws-sdk-net/issues/1872#issuecomment-863455320, which has a note that per https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Scan.html, the ScanFilter is a legacy parameter. Recommended technique is to use Filter Expression, example at https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LowLevelDotNetScanning.html.
Thanks,
Ashish


Identifiers validation failed for dataset Inventario de gasterópodos terrestres y dulceacuícolas del área focal de Ixcan, Chiapas

Identifier validation failed for the dataset Inventario de gasterópodos terrestres y dulceacuícolas del área focal de Ixcan, Chiapas:

Crawler attempt: 268
Publishing organization: Comisión nacional para el conocimiento y uso de la biodiversidad
Publishing country: MEXICO
Installation: IPTCONABIO
Number of occurrences indexed for this dataset at the time of the issue: 277
Cause: GBIF IDs hit maximum allowed - 50%, duplicates - 100%, total records count 277, absent records count 277
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/3073304/step

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the Inventario de gasterópodos terrestres y dulceacuícolas del área focal de Ixcan, Chiapas : https://doi.org/10.15468/4wpadv.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.

Identifier validation still  failing for the dataset Inventario de gasterópodos terrestres y dulceacuícolas del área focal de Ixcan, Chiapas:

Crawler attempt: 271
Publishing organization: Comisión nacional para el conocimiento y uso de la biodiversidad
Publishing country: MEXICO
Installation: IPTCONABIO
Number of occurrences indexed for this dataset at the time of the issue: 277
Cause: GBIF IDs hit maximum allowed - 50%, duplicates - 100%, total records count 277, absent records count 277
Pipelines execution steps: http://api.gbif.org/v1/pipelines/history/execution/3094128/step

Publisher email
Hello,
I am contacting you from the GBIF Secretariat about a dataset published by the Inventario de gasterópodos terrestres y dulceacuícolas del área focal de Ixcan, Chiapas : https://doi.org/10.15468/4wpadv.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.
As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.
In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.
We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.
Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?
Please let us know if you have any question. Thanks!
All the best,

You can skip/fix identifier validation using the registry UI.


read_deltalake on Unity Catalog Table from Databricks has invalid region configuration

I am trying to read a table stored in Unity Catalog (external data access enabled) in Databricks and I am getting "OSError: Generic S3 error: Received redirect without LOCATION, this normally indicates an incorrectly configured region", even though the region is explicitly defined in io_config:
import daft
from daft.unity_catalog import UnityCatalog
from daft.io import IOConfig, S3Config
from dotenv import dotenv_values

env_cfg = dotenv_values()

unity = UnityCatalog(
    endpoint=env_cfg.get('DBX_ENDPOINT'),
    token=env_cfg.get('DBX_TOKEN'),
)

print(unity.list_catalogs())# See all available catalogs, works OK
print(unity.list_schemas('test_catalog')) # See available schemas in a given catalog, works OK
print(unity.list_tables('test_catalog.test_schema')) # See available tables in a given schema, works OK

cfg = unity.load_table('test_catalog.test_schema.test_table') # works OK
io_config = IOConfig(s3=S3Config(region_name='eu-central-1'))
cfg_df = daft.read_deltalake(cfg, io_config=io_config) # here is, where the error happens

And I am getting this output
[...catalogs...]
[...schemas...]
[...tables...]
With this error
failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
S3 Credentials not provided or found when making client for us-east-1! Reverting to Anonymous mode. the credential provider was not enabled
[2024-09-23T13:10:00Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
[2024-09-23T13:10:00Z WARN  aws_config::imds::region] failed to load region from IMDS err=failed to load IMDS session token: dispatch failure: io error: error trying to connect: tcp connect error: Connection refused (os error 111): tcp connect error: Connection refused (os error 111): Connection refused (os error 111) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 111, kind: ConnectionRefused, message: "Connection refused" })), connection: Unknown } }) }))
Traceback (most recent call last):
File "poc_uc_daft.py", line 29, in
cfg_df = daft.read_deltalake(cfg, io_config=io_config)
File "/c/Users/xxx/Projects/xxx/venv/lib/python3.8/site-packages/daft/api_annotations.py", line 39, in _wrap
return timed_func(*args, **kwargs)
File "/c/Users/xxx/Projects/xxx/venv/lib/python3.8/site-packages/daft/analytics.py", line 228, in tracked_fn
result = fn(*args, **kwargs)
File "/c/Users/xxx/Projects/xxx/venv/lib/python3.8/site-packages/daft/io/_deltalake.py", line 74, in read_deltalake
delta_lake_operator = DeltaLakeScanOperator(table_uri, storage_config=storage_config)
File "/c/Users/xxx/Projects/xxx/venv/lib/python3.8/site-packages/daft/delta_lake/delta_lake_scan.py", line 63, in init
self._table = DeltaTable(
File "/c/Users/xxx/Projects/xxx/venv/lib/python3.8/site-packages/deltalake/table.py", line 380, in init
self._table = RawDeltaTable(
OSError: Generic S3 error: Received redirect without LOCATION, this normally indicates an incorrectly configured region
Desktop (please complete the following information):

Windows 11

Am doing something wrong or is it a bug? Is there a workaround?
May it be related to this 2 days old issue? https://github.com/Eventual-Inc/Daft/issues/2879

👁 @kevinzwang

Yeah the delta-rs SDK is very dumb about regions -- we have to provide it with exactly the right region otherwise it will freak out.
Reading the code, we take the IOConfig (and hence the region) provided by Unity Catalog:
        # Override the storage_config with the one provided by Unity catalog
        table_io_config = table.io_config
        if table_io_config is not None:
            storage_config = StorageConfig.native(NativeStorageConfig(multithreaded_io, table_io_config))

My guess is that Unity doesn't give us the right region, or likely doesn't give us a region at all. We might need to corner case this 😬

opt. B - it's not giving any region.

I checked it and my Databricks / Unity Catalog instance is definitely on "eu-central-1" as stated in the code above.
I printed cfg.io_config (unity catalog table . io config) and its None

It looks like we currently ignore the given io_config if you pass in a unity catalog table, and we do not construct an S3Config with a region in our UnityCatalog.load_table.
We'd like to get a fix for this but in the mean time @lukaskratoch what you can do is add the region to the table's io config:
cfg = unity.load_table('test_catalog.test_schema.test_table')
cfg.io_config = IOConfig(s3=cfg.io_config.s3.replace(region_name='eu-central-1'))
cfg_df = daft.read_deltalake(cfg) # should work now

I am afraid that workaround will not work:

Initially I was getting this err:

UnityCatalogTable has to be set to frozen=False (I just overwrote it in local libraries)
After that I am getting this err:
DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 30s }), connection: Unknown } })
which I guess reffers to this issue in delta-rs

@lukaskratoch what version of deltalake are you using?
As for a solution to the io config thing without having to modify the library, you could perhaps extract the table_uri and the io_config and pass those in manually.
cfg = unity.load_table('test_catalog.test_schema.test_table')
io_config = IOConfig(s3=cfg.io_config.s3.replace(region_name='eu-central-1'))
daft.read_deltalake(cfg.table_uri, io_config=io_config)

@kevinzwang , your recommended solution that did the trick for me. I had run into the same issue that @lukaskratoch had run into as well and was searching if an issue was logged on this and found this.
@kevinzwang ,
A question I had, which I couldn't figure out yet is if there is any way that APIs from daft could figure out the region of the table URI ?
If it is not possible, I think the Databricks SDK WorkspaceClient might be a way to get the region of the metastore to pass it down to daft.read_deltalake
this is the piece of code that works for me and uses Databricks SDK WorkspaceClient  to retrieve the metastore region.
import daft # Using version 0.3.4
from daft.unity_catalog import UnityCatalog
from daft.io import IOConfig, S3Config
from databricks.sdk import WorkspaceClient # Using version  0.33.0
import os

DATABRICKS_HOST = os.environ.get('DATABRICKS_HOST')
PAT_TOKEN_AWS = os.environ.get('PAT_TOKEN_AWS') #i.e, the PAT token stored in environment variables

w = WorkspaceClient(host=DATABRICKS_HOST, token=PAT_TOKEN_AWS)
metastore_summary = w.metastores.summary()

unity = UnityCatalog(endpoint=DATABRICKS_HOST,token=PAT_TOKEN_AWS)

# Read an external table
unity_table_ext = unity.load_table("main_catalog.cust_another_schema.sample_external_access_table") # This is an external table
io_config = IOConfig(s3=unity_table_ext.io_config.s3.replace(region_name=metastore_summary.region))
df_ext = daft.read_deltalake(unity_table_ext.table_uri,io_config=io_config)
df_ext.show()

@kevinzwang

@lukaskratoch what version of deltalake are you using?

previously deltalake==0.19.1, today I updated to deltalake==0.20.1, still having the same connection timeout "DispatchFailure" error.

@lukaskratoch @anilmenon14 Thank you for the information! I am taking a look into these issues and hope to have an update for you soon

Adding an azure perspective here, this is what I had to do to be able to read from the table, which seems like credential vending just isn't working for azure since it's still trying s3.
from daft.unity_catalog import UnityCatalog
import daft
from deltalake.table import DeltaTable
from daft.io import IOConfig, AzureConfig

azure_config = AzureConfig(
    storage_account="myaccount",
    tenant_id="my_tenant",
    client_id="my_client",
    client_secret="my_secret",
)

io_config = IOConfig(azure=azure_config)

unity = UnityCatalog(
    endpoint="https://<workspace-address>.azuredatabricks.net",
    token="my_token",
)

unity_table = unity.load_table("catalog.schema.table")
df = daft.read_deltalake(unity_table.table_uri, io_config=io_config)

df.show()

It still gives me the output of:
failed to load region from IMDS err=failed to load IMDS session token: dispatch
failure: io error: error trying to connect: tcp connect error: A socket operation was attempted to an unreachable network. (os error 10051): tcp connect error:
A socket operation was attempted to an unreachable network. (os error 10051): A
socket operation was attempted to an unreachable network. (os error 10051) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 10051, kind: NetworkUnreachable, message: "A socket operation was attempted to an unreachable network." })), connection: Unknown } })
}))
failed to load region from IMDS err=failed to load IMDS session token: dispatch
failure: io error: error trying to connect: tcp connect error: A socket operation was attempted to an unreachable network. (os error 10051): tcp connect error:
A socket operation was attempted to an unreachable network. (os error 10051): A
socket operation was attempted to an unreachable network. (os error 10051) (FailedToLoadToken(FailedToLoadToken { source: DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 10051, kind: NetworkUnreachable, message: "A socket operation was attempted to an unreachable network." })), connection: Unknown } })
}))
S3 Credentials not provided or found when making client for us-east-1! Reverting to Anonymous mode. the credential provider was not enabled

Didn't seem like anything I tried with the s3 config part did anything. I had to pass in my own credentials entirely.

Hi @jordandakota ,
I have been running into the same problem on Azure storage and logged a new issue 3024.
I believe I know where the issue is and will raise a PR this week with a solution to try to get Azure support.

Yes, when we first built this integration the Unity Catalog only vended S3 credentials 😛
I think unity has made some progress since then, but we actually do need to probably upgrade the Python SDK to get the new updated API spec.
WRT regions, we will have to play around with the API a little to figure out what databricks' implementation of Unity is returning us.

v0.3.9 has just been released which should fix the S3 region issue! @lukaskratoch @anilmenon14 please try it out and let us know how it goes
@jordandakota your Azure issue was also addressed in the latest release. If you have trouble with it, feel free to create a new issue to report it

@kevinzwang, Thanks for the update on the S3 region issue fix. I can confirm it works fine, but with a small issue. It appears botocore and boto3 are not part of the dependencies to install and hence are not installed as default. I installed them separately in my virtual environment and got past the issue.
I can confirm this pattern now works for AWS, without having to pass down the region name for AWS regions outside us-east-1
unity = UnityCatalog(endpoint=DATABRICKS_HOST,token=PAT_TOKEN_AWS)
unity_table_mngd = unity.load_table("some_catalog.some_schema.some_table")  # This table storage is on eu-west-1
df_mngd = daft.read_deltalake(unity_table_mngd)
df.show()

As for the Azure issue, this appears to be fixed too, but needs a bit of workaround which I am hoping we can avoid in the future.
What does not work for Azure:
unity = UnityCatalog(endpoint=DATABRICKS_HOST,token=PAT_TOKEN_AZURE)
unity_table_ext = unity.load_table("ucx_migrated.ucx_dst_external.external_dbfs_mounted") # This is an external table
df_ext = daft.read_deltalake(unity_table_ext)
df_ext.show()

Error:
DaftCoreException: DaftError::External Generic AzureBlob error: Azure Storage Account not set and is required.
 Set either `AzureConfig.storage_account` or the `AZURE_STORAGE_ACCOUNT` environment variable.

**What works for Azure now **
unity = UnityCatalog(endpoint=DATABRICKS_HOST,token=PAT_TOKEN_AZURE)
unity_table_ext = unity.load_table("ucx_migrated.ucx_dst_external.external_dbfs_mounted") # This is an external table

regex_match_az_storage_acc = re.search(r'@([^\.]+)\.', unity_table_ext.table_uri) # Gather storage account name from table URI
if regex_match_az_storage_acc:
    storage_account_parsed = regex_match_az_storage_acc.group(1)
else:
    raise ValueError("{} does not appear to be a valid Azure Storage URI".format(unity_table_ext.table_uri))

io_config = IOConfig(azure=unity_table_ext.io_config.azure.replace(storage_account=storage_account_parsed))
df_ext = daft.read_deltalake(unity_table_ext.table_uri,io_config=io_config)
df_ext.show()

@jordandakota , when you have a chance, you can test as well if you see this behavior in Azure.
@kevinzwang , I haven't had a chance to review the internals of df.show() to inspect this behavior in Azure and we could try and solve it from a new issue logged, if you agree.

I did test and came to the same workaround just slightly differently. Reporting in now as confirmed.

Hi @kevinzwang , As @anilmenon14 mentioned i too got the same error for Azure. Replacing the storage account name solved the issue for me. Please note or create a new issue, as this seems to be a workaround.
Works in Azure:
io_config = IOConfig(azure=unity_table_ext.io_config.azure.replace(storage_account='storage_account'))
df = daft.read_deltalake(table=unity_table_ext.table_uri,io_config=io_config)
df.show()

@anilmenon14 @g-kannan will take a look at these issues, thank you for reporting them. Tracking it in a new issue  #3142


pickle issue with CCL in emcee

This comes courtesy of Kai Hoffmann, which appeared when using CCL objects in multiprocessing in emcee. Relevant code to follow.
Error message:
AttributeError                            Traceback (most recent call last)
 in
----> 1 fit_wgp(corr_rcut, p_cosmo, p_bias, p_nla_ini, fname_chain, redshift, 'nla')
 in fit_wgp(data, p_cosmo, p_bias, p_IA_ini, fname_chain, redshift, IA_model)
91             sampler.run_mcmc(initial, nstep, progress=True)
92
---> 93     run_mcmc(new_chain, nwalkers, ncore, nstep, fname_chain)
94
95     return 0
 in run_mcmc(new_chain, nwalkers, ncore, nstep, fname_chain)
89             )
90
---> 91             sampler.run_mcmc(initial, nstep, progress=True)
92
93     run_mcmc(new_chain, nwalkers, ncore, nstep, fname_chain)
~/.local/lib/python3.8/site-packages/emcee/ensemble.py in run_mcmc(self, initial_state, nsteps, **kwargs)
382
383         results = None
--> 384         for results in self.sample(initial_state, iterations=nsteps, **kwargs):
385             pass
386
~/.local/lib/python3.8/site-packages/emcee/ensemble.py in sample(self, initial_state, log_prob0, rstate0, blobs0, iterations, tune, skip_initial_state_check, thin_by, thin, store, progress)
283             state.blobs = blobs0
284         if state.log_prob is None:
--> 285             state.log_prob, state.blobs = self.compute_log_prob(state.coords)
286         if np.shape(state.log_prob) != (self.nwalkers,):
287             raise ValueError("incompatible input dimensions")
~/.local/lib/python3.8/site-packages/emcee/ensemble.py in compute_log_prob(self, coords)
425                 map_func = map
426             results = list(
--> 427                 map_func(self.log_prob_fn, (p[i] for i in range(len(p))))
428             )
429
/usr/lib/python3.8/multiprocessing/pool.py in map(self, func, iterable, chunksize)
362         in a list that is returned.
363         '''
--> 364         return self._map_async(func, iterable, mapstar, chunksize).get()
365
366     def starmap(self, func, iterable, chunksize=None):
/usr/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769             return self._value
770         else:
--> 771             raise self._value
772
773     def _set(self, i, obj):
/usr/lib/python3.8/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
535                         break
536                     try:
--> 537                         put(task)
538                     except Exception as e:
539                         job, idx = task[:2]
/usr/lib/python3.8/multiprocessing/connection.py in send(self, obj)
204         self._check_closed()
205         self._check_writable()
--> 206         self._send_bytes(_ForkingPickler.dumps(obj))
207
208     def recv_bytes(self, maxlength=None):
/usr/lib/python3.8/multiprocessing/reduction.py in dumps(cls, obj, protocol)
49     def dumps(cls, obj, protocol=None):
50         buf = io.BytesIO()
---> 51         cls(buf, protocol).dump(obj)
52         return buf.getbuffer()
53
AttributeError: Can't pickle local object 'fit_wgp..log_probability'

Is the log_probability function here defined by some chance within the fit_wgp function?  That could explain the pickling issue; might not have anything to do with CCL.  Could you post more of what fit_wgp() looks like?

Hi @timothydmorton. Indeed, log_probability is defined inside fit_wgp. The code from my colleague is pasted below.

def fit_wgp(nwalkers, ncore, nstep, ndim, a1_ini):
zz_loc = 0.5
cosmo_loc = ccl.Cosmology(Omega_c=0.27, Omega_b=0.045, h=0.67, A_s=2.1e-9, n_s=0.96)
ptc_loc = pt.PTCalculator(with_NC=True, with_IA=True, log10k_min=-5, log10k_max=2, nk_per_decade=20)
ptt_g_loc = pt.PTNumberCountsTracer(b1=1, b2=0, bs=0)

def log_likelihood_loc(a1):

    rp_model, wgp_model = get_wgp_nla(cosmo_loc, zz_loc, ptc_loc, ptt_g_loc, a1)

    wg_model_interpol = np.interp(data_glob.rp, rp_model, wgp_model)

    delta = (data_glob.wgp - wg_model_interpol)/data_glob.wgperr

    ll = -0.5 * np.dot(delta, delta)

    if np.isnan(ll):
        return -np.inf
    return ll

def log_prior_loc(a1):
    if -10 < a1 < 20:
        return 0.0
    return -np.inf

def log_probability_loc(p):

    a1 = p[0]

    lp = log_prior_loc(a1)

    if not np.isfinite(lp):
        return -np.inf
    return lp + log_likelihood_loc(a1)

a1_ini = 1.
initial = a1_ini + 0.1 * np.random.randn(nwalkers, ndim)

with Pool(ncore) as pool:
    sampler = emcee.EnsembleSampler(
    nwalkers,
    ndim,
    log_probability_loc,
    moves=[(emcee.moves.DEMove(), 0.8), (emcee.moves.DESnookerMove(), 0.2)],
    pool=pool
    )

    sampler.run_mcmc(initial, nstep, progress=True)

This won't work because the transiently-defined function is not able to be pickled, and the parallelization within emcee happens by sending pickled objects around between processes.  The solution would be to refactor this code to make sure the log_prob function is defined at the outer level of the script, not within the function.

To be more specific, if the entire fit_wgp were not a function but just a standalone script (e.g., everything currently defined within the function just moved to the outermost level of the .py file, and the function arguments either hardcoded or taken in via argparse), the above code should work as is.

Thanks @timothydmorton ! Just to be sure, this has nothing to do with CCL? I will close the issue and pass this along.

Correct.  To be clear, I don't know whether pickling might be an issue with CCL in general or not, but I know that the above would raise an error with emcee whatever the guts of the likelihood.  If a pickling error still shows up after moving everything to the top level, then you could re-open.


perf: stale page check

When the user scrolls, a scroll event listener starts a process to check if cells that are scrolled into view are missing data. When this is the case a request to fetch data for those missing cells is sent to Engine. But sometimes the user scrolls rather quickly, so quickly that some of the cells that where empty and needed data, are no longer in the visible view of the user.
This PR attempts to optimize that process, so that when a cell is no longer visible to the user and the data has not already been fetch. It's considered stale and not included in the next data fetch.

Looking through my review requests and found this. Should I take a look?

Looking through my review requests and found this. Should I take a look?

I think you can wait. It will take a look at the "note to self" when I get the time. It should still be valid.

Closing, lets do something similar to PVT where fetching is throttled/debounced and uses two buffers


md-select cannot be placed inside md-input-container, hence labels position are not same as in an md-input

Bug, feature request, or proposal:
Bug
What is the expected behavior?
material breaks rendering the form if contains an md-select inside an md-input-container
This code will print the label on top of the md-input
    <md-input-container>
            <label for="firstName">First Name</label>
            <input mdInput id="firstName" name="firstName" autocomplete="off"
                   formControlName="firstName">

            <md-error *ngIf="formErrors.firstName">
              {{ formErrors.firstName }}
            </md-error>
    </md-input-container>

md-select element behaves differently as it can't be placed inside a md-input-container (labels showed on the left side of the select element instead ob being on top).
What is the current behavior?
labels assigned to an md-select behaves differently as the ones assigned to an md-input inside an md-input-container
What are the steps to reproduce?
Providing a Plunker (or similar) is the best way to get the team to see your issue.
Plunker template: http://embed.plnkr.co/sCUAxJWox2BG1p5hD1fU/
What is the use-case or motivation for changing an existing behavior?
improve material 2 functionality
Which versions of Angular, Material, OS, TypeScript, browsers are affected?
@angular/cli: 1.3.1
node: 7.9.0
os: win32 x64
@angular/animations: 4.3.5
@angular/cdk: 2.0.0-beta.8
@angular/common: 4.3.5
@angular/compiler: 4.3.5
@angular/core: 4.3.5
@angular/forms: 4.3.5
@angular/http: 4.3.5
@angular/material: 2.0.0-beta.8
@angular/platform-browser: 4.3.5
@angular/platform-browser-dynamic: 4.3.5
@angular/router: 4.3.5
@angular/cli: 1.3.1
@angular/compiler-cli: 4.3.5
@angular/language-service: 4.3.5
Is there anything else we should know?

There is active work to be able to use the select inside the input-container. Since the last release, the input container has been renamed to md-form-field and https://github.com/angular/material2/pull/6488 is aligning the select's behavior/styles with the input.

What @willshowell said. We're tracking this via #2124