⚠ Home NAS has degraded performance

In c1d8f31, Home NAS ($MY_NAS_URL) experienced degraded performance:

HTTP code: 200
Response time: 18578 ms

Resolved: Home NAS performance has improved in 25c9dea after 46 minutes.


[source-kafka] Fix error handling for kubeconfig

see https://github.com/knative/client-contrib/pull/33#discussion_r446014741

/assign


Compilation installation issues

mypyc ../speedup.py ..\speedup.py:13: error: Skipping analyzing "mypyc.build": module is installed, but missing library stubs or py.typed marker ..\speedup.py:13: note: See https://mypy.readthedocs.io/en/stable/running_mypy.html#missing-imports
Don't know why give me a hint of this, I followed the mypyc official documentation also does not work

Can I write a compilation tutorial?

Just run the following command
https://github.com/abersheeran/baize/blob/cb03a8684358a9ae2c2a412440abe71fa8e821cc/.github/workflows/ci.yml#L61


Non-descriptive error "body used already for:" when using the odata handler to talk to Microsoft Graph

When using the odata handler to talk to Microsoft Graph, if the bearer token is not specific, is invalid, or if the access token expires, the following error is returned
{
  "errors": [
    {
      "message": "body used already for: ",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "me"
      ]
    }
  ],
  "data": {
    "me": null
  }
}

It would be helpful if there was a more descriptive message like 'Unauthorized" assuming the API returns a 401.

Available in 0.2.15!

Sweet!


Multiple node processes support with a single datastore file

I was running two node servers, each on different ports, but both reading and writing to the same persistent datastore file. It looks like each node process is reading its own copy of the doc from the file.
For example, node process one creates the first record below, and second node process creates the second record. The dump of the data file contains both record, but each process can only read one that has been created by itself.
{"path":"/path","time":1386333352657,"_id":"FLRlkL2deQMA1RyZ"}
{"path":"/path","time":1386333358367,"_id":"BzTn0GRGq67Mwco8"}

Closing as no activity

If anyone's interested, I'm trying to do the same for nedb as level-party does for leveldb:
https://github.com/allain/nedb-party
It uses rpc to send requests to the process that "owns" the db. It's still early days, but I hope to be committing the time to make this robust.

Recently I struggled with this problem and came up with a solution that is similar to nedb-party but a bit more robust: https://github.com/vangelov/nedb-multi, hope it helps someone.


delete-translation URL broken?

Hi all,
When I try to delete a translation in the admin I get a 404. Turns out the URL:
/en/admin/events/event/2/change/delete-translation/en/
is wrong.
When I remove the "/change" from it, it works:
/en/admin/events/event/2/delete-translation/en/
Is this a bug?
I am running
Django==1.9.2
django-parler==1.6.1
django-cms==3.2.1

Django 1.9 changed the path to the change form from /admin/app/model/id/ to /admin/app/model/change/id/
Probably the best option is to register both versions as catch all

Hi! Could you provide me with some pointers as to how I would do that?

@maltebeckmann I actually meant the option for parler to do this.
Declare a get_urls in your admin (like in https://github.com/edoburu/django-parler/blob/master/parler/admin.py#L308)
And add something like:
        return [
            url(r'^change/(.+)/delete-translation/(.+)/$',
                self.admin_site.admin_view(self.delete_translation),
                name='{0}_{1}_change_delete_translation'.format(*info)
                ),
        ] + urlpatterns

@maltebeckmann see my PR above

@yakky Thanks - still getting my head around Django...


Getting started guide improvements

In step 6, deploy skipper, there is a notice that one should uncomment properties in src/kubernetes/server/server-deployment.yaml  This should instead be described in the next step.
Step 7.  Deploy the Data Flow Server and  Step 8. Deploy the Spring Cloud Data Flow Server for Kubernetes... are redundant titles
These steps be combined or Step 7 should be about additional modifications to the yml files, while step 8 is just about shipping the yaml files to k8s.

Add  kubectl get all -l app=skipper in the skipper section so the user can do some basic sanity checks.

There are two sections for installing skipper, there should be only one.  The second section seems to be the correct one.  The section on installing skipper should come before the section on installing the scdf server

Wrong url addresses for streams deployed using skipper section.  Says
server-unknown:>dataflow config server http://dataflow-server.cfapps.io
Successfully targeted http://dataflow-server.cfapps.io
dataflow:>

this should reference some generic GKE address or a minikube address.

Add the metadata url when registering or at least mention it.  When executing the command
Information about source application 'time':
Version: '1.2.0.RELEASE':
Default application version: 'true':
Resource URI: docker:springcloudstream/time-source-rabbit:1.2.0.RELEASE
╔══════════════════════════════╤══════════════════════════════╤══════════════════════════════╤══════════════════════════════╗
║         Option Name          │         Description          │           Default            │             Type             ║
╚══════════════════════════════╧══════════════════════════════╧══════════════════════════════╧══════════════════════════════╝

the table will be down in the shell, but it isn't listed in the docs.  This may cause some confusion.  Maybe instead of an empty table, it should say 'no application metadata jar available, application properties are not available for listing'


dyld: Library not loaded realm

Bugs

i added a framework that i developed in ios to the new project. I tried to make the app universal (one framework work both in device and simulator) according to this tutorial create an ios universal framework . When i run my demo project using this universal framework i get this errors
dyld: Library not loaded: @rpath/Realm.framework/Realm
Referenced from: /private/var/mobile/Containers/Bundle/Application/DDF71B22-F535-43E5-B770-D3425419B108/DemoSDk2.app/Frameworks/#######.framework/######### Reason: no suitable image found. Did find: /private/var/mobile/Containers/Bundle/Application/DDF71B22-F535-43E5-B770-D3425419B108/DemoSDk2.app/Frameworks/########.framework/Frameworks/Realm.framework/Realm: mmap() errno=1 validating first page of '/private/var/mobile/Containers/Bundle/Application/DDF71B22-F535-43E5-B770-D3425419B108/DemoSDk2.app/Frameworks/##########.framework/Frameworks/Realm.framework/Realm'
Version of Realm and Tooling

Realm framework version: 2.0
Realm Object Server version: ?
Xcode version: 7.3
iOS/OSX version: 10.12
Dependency manager + version: ?

@sassiwalid Maybe you forgot to add the parent path of RealmSwift.framework to the “Framework Search Paths” in build settings. Because RealmSwift.framework uses Realm.framework internally, so RealmSwift.framework should be able to find Realm.framework and linked to.
FYI: Unfortunately AppStore rejects if your app includes such nested frameworks. In other words, even if your attempt succeeds, no one can distribute apps that use the framework to AppStore. That is why RealmSwift.framework does not include Realm.framework.

thank you, sure when i send my framework to the appstore, it's was rejected. so i decided to make realm outside of my framework. Best regards

I had the same issue in Xcode 11.4 and I fixed it by editing my pod file to:
platform :ios, '13.0'
target 'YourTarget' do
#use_frameworks! -> !!! very important
pod 'Realm', :modular_headers => true
pod 'RealmSwift', :modular_headers => true
end


SymbolPackageFormat

Hi,
What's the difference between what's explain in this page and what's explained here?
I find it difficult to understand whether the snupkg symbol package format is the newer one.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 98435fc3-0e6f-91e8-b2ba-5329fffd193b
Version Independent ID: ea0fe9fc-d171-8ddf-426d-a7ec3a232445
Content: How to create NuGet symbol packages
Content Source: docs/create-packages/Symbol-Packages.md
Product: nuget
GitHub Login: @karann-msft
Microsoft Alias: karann

Hello! To answer your question, you should use the .snupkg format if you can. There's a little note on the documentation that you linked:

The new recommended format for symbol packages is .snupkg. See Creating symbol packages (.snupkg).
.symbols.nupkg is still supported but only for compatibility reasons.

It seems that this note is commonly missed. Is it possible to make it more eye catching? @karann-msft @Mikejo5000 Any suggestions?
In any case, I've opened a change to make it harder to accidentally find the legacy symbol documentation: https://github.com/NuGet/docs.microsoft.com-nuget/pull/1724.

My bad. I guess I went right to the topic and missed it.
I guess this issue can be closed. Sorry and thanks.

No worries! You're not the first person to make that mistake, it seems that our docs should be clearer

Is this still a case with sourcelink and lack of support for snupkg in azure devops?

Is this still a case with sourcelink and lack of support for snupkg in azure devops?

Hello, Azure Artifacts does not support .snupkg symbol packages yet. They recommend pushing symbols using the PublishSymbols@2 task. See: https://developercommunity.visualstudio.com/comments/1247260/view.html

Hello, Azure Artifacts does not support .snupkg symbol packages yet. They recommend pushing symbols using the PublishSymbols@2 task. See: https://developercommunity.visualstudio.com/comments/1247260/view.html


Create a logo for Scrollodex

i can try this one

hi @neeraj15022001 - as i commented before, this isn't ready to be worked on yet. I will add a label to the issues.

Not a problem i will reach back again later

Display Logo in the header, if no header exists yet or at the time of your PR, add the logo to a design folder in the repo.

You can assisgn it to anyone else, thanks for asking

Can I work on this? I'm not sure how to add to a header without breaking things 😅 but I can definitely add a few logo concepts into a folder :) @domsbytes

Can I work on this? I'm not sure how to add to a header without breaking things 😅 but I can definitely add a few logo concepts into a folder :) @domsbytes

@Kitkatnik Yes! Sounds great.

@jischr @domsbytes I see a public folder with the WWCBD logo. Do you want me to put it in there? Or create a design folder within public?

@Kitkatnik you can put them in there!

hey, I'd like to work on this, kindly assign me this issue.


Dynamic recompilation fails when a class uses a system diagnostic attribute

OS: Kubuntu 21.10
.NET versions: Confirmed on both 5.0.404 and 6.0.200
osu-framework: 2022.214.0-ish
When I use an attribute from the System.Diagnostics.CodeAnalysis namespace, e.g. NotNullIfNotNullAttribute, an error similar to this occurs during dynamic recompilation (reformatted for sanity):
System.AggregateException: One or more errors occurred.
(/osu.Framework/Graphics/Visualisation/VisualisedDrawable.cs(1161,10): error CS0433: The type 'NotNullAttribute' exists in both
'Microsoft.Build.Framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' and
'Newtonsoft.Json, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed')
(/osu.Framework/Graphics/Visualisation/VisualisedDrawable.cs(269,18): error CS0433: The type 'NotNullIfNotNullAttribute' exists in both
'Microsoft.Build.Framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' and
'nunit.framework, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=2638cd05610744eb')

I tried using a similar solution by banning assemblies other than System.Private.CoreLib that define System.* types, but then NUNit framework wasn't loaded and unit test failed recompilation


Pending rewards flickers on optimism

For networks with the gauge reward helper, it tries to preserve reward amounts on polling refetches to avoid the flickering. needs to be tested.

apparently the new RPC did not resolve this issue.

also please test and try this RPC node: https://rpc.ankr.com/optimism


crf-search panics almost immediately on any file

Hello,
I've installed ab-av1 using cargo install --git https://github.com/alexheretic/ab-av1 to make sure I have the latest version. When I try running the command ~/.cargo/bin/ab-av1 crf-search -i test.mp4 --preset 6 the task panics after 10-40 seconds on any video file.
Here is the error with the stack trace:
thread 'main' panicked at 'can not convert float seconds to Duration: value is either too big or NaN', /builddir/build/BUILD/rustc-1.65.0-src/library/core/src/time.rs:736:23
stack backtrace:
   0:     0x55c792d5c889 - <unknown>
   1:     0x55c792d08dae - <unknown>
   2:     0x55c792d335c4 - <unknown>
   3:     0x55c792d5d813 - <unknown>
   4:     0x55c792d5e74d - <unknown>
   5:     0x55c792d5e312 - <unknown>
   6:     0x55c792d5e27c - <unknown>
   7:     0x55c792d5e251 - <unknown>
   8:     0x55c792c66e42 - <unknown>
   9:     0x55c792c80936 - <unknown>
  10:     0x55c792c8f2c1 - <unknown>
  11:     0x55c792cab021 - <unknown>
  12:     0x55c792c873ef - <unknown>
  13:     0x55c792cb6786 - <unknown>
  14:     0x55c792ca6560 - <unknown>
  15:     0x55c792cb5aa7 - <unknown>
  16:     0x7f98b7cc7510 - __libc_start_call_main
  17:     0x7f98b7cc75c9 - __libc_start_main_alias_2
  18:     0x55c792c6a195 - <unknown>
  19:                0x0 - <unknown>

I'm running Fedora 37, if that's important.

Running sample-encode throws the same error, so it looks like it's a similar issue to #8.

Just tested the 0.5.0 release and it works fine.

Thanks for the report. Since v0.5.0 we started ffprobing the encoded samples, so it looks like they have an invalid duration for you.
I've pushed a fix to main that should avoid this panicking b4acad00e9b9fa9fb620d20e78e66bc3018e0a0a. Can you re-try?

Nope, the result is the same.
thread 'main' panicked at 'can not convert float seconds to Duration: value is either too big or NaN', /builddir/build/BUILD/rustc-1.65.0-src/library/core/src/time.rs:736:23
stack backtrace:
   0:     0x559001e189e9 - <unknown>
   1:     0x559001dc4e4e - <unknown>
   2:     0x559001def654 - <unknown>
   3:     0x559001e19973 - <unknown>
   4:     0x559001e1a8ad - <unknown>
   5:     0x559001e1a472 - <unknown>
   6:     0x559001e1a3dc - <unknown>
   7:     0x559001e1a3b1 - <unknown>
   8:     0x559001d22e82 - <unknown>
   9:     0x559001d46a37 - <unknown>
  10:     0x559001d45bc2 - <unknown>
  11:     0x559001d61c49 - <unknown>
  12:     0x559001d3dbf4 - <unknown>
  13:     0x559001d738a6 - <unknown>
  14:     0x559001d5cfc3 - <unknown>
  15:     0x559001d72bc7 - <unknown>
  16:     0x7f08f0360510 - __libc_start_call_main
  17:     0x7f08f03605c9 - __libc_start_main_alias_2
  18:     0x559001d261d5 - <unknown>
  19:                0x0 - <unknown>

Thanks for retesting.
Can you re-build with RUSTFLAGS='-C strip=none' cargo install --git https://github.com/alexheretic/ab-av1 so the stack trace will have all the symbols. That could be helpful for zeroing in on what's panicking.

I've had another look and this could possibly be caused by, rather than an invalid, a zero ffprobe duration of the encoded samples. I can fix that scenario by ignoring zero in that case, please try the latest commit bc2bd5d.

It works now. Thank you, wizard!


Would you like a script that runs the pretrained models?

I made one yesterday, and I was wondering if you'd like me to open a PR so that we can add it to the project? That way, people could run a simple command line instead of writing their own scripts or doing it in REPL. Let me know, and I'll open a PR if you think that makes sense.

Would be nice.

That'd be great!


Query Clause Level Stats in OpenSearch

Is your feature request related to a problem? Please describe.
With the release of Neural Search plugin we release added a new query clause called as "neural". The query clause takes up a query text and model_id and re-writes the query to a k-NN based query. We want to add the stats for this new query clause in OpenSearch which will provide information like: how many times this query clause was invoked, latency, errors etc.
This is a problem for all the query clauses and not the neural query clause. Hence the solution should be OpenSearch specific and not plugin specific.
Describe the solution you'd like
Given that currently we don't have any mechanism to track the metrics described above for all the different query clauses, solution is to implement a stats api that can provide this information.
Describe alternatives you've considered
The only alternative that I can think of is plugins create a new profile/stats api that can provide stats for the query clauses. But this will make every plugin to reinvent the wheel and won't be consistent across different query clauses.
The Stats api present in OpenSearch provide results for the search api combined but it doesn't give metrics around specific query clauses.
GET _stats?pretty&filter_path=_all.primaries.search

Response

{
  "_all" : {
    "primaries" : {
      "search" : {
        "open_contexts" : 0,
        "query_total" : 1779,
        "query_time_in_millis" : 407,
        "query_current" : 0,
        "fetch_total" : 1779,
        "fetch_time_in_millis" : 763,
        "fetch_current" : 0,
        "scroll_total" : 1,
        "scroll_time_in_millis" : 58,
        "scroll_current" : 0,
        "point_in_time_total" : 0,
        "point_in_time_time_in_millis" : 0,
        "point_in_time_current" : 0,
        "suggest_total" : 0,
        "suggest_time_in_millis" : 0,
        "suggest_current" : 0
      }
    }
  }
}

Additional context

Enabling this metrics will allow customer to understand which are the most used query clauses and will also help them to understand the query patterns. This will also provide a consistent experience for OpenSearch Customers for understanding the query.
For plugin owners who are adding new query clauses it will reduce the work for adding new metrics/rest handlers for the stats.

@nknize @dblock can we build this feature in OpenSearch core. This can avoid a lot of duplicate work which plugins has to do.

At a high level this seems like a good idea to me, provided it's very generic.

At a high level this seems like a good idea to me, provided it's very generic.

@dblock Can we core team prioritize this in their roadmap?

@navneet1v I doubt it AFAIK they are focused on storage and extensibility, but feel free to contribute!

@dblock @navneet1v I'd like to keep the Core focus on storage and extensibility unless we have volumes of community members asking for this, or as @dblock feel free to contribute!

Thanks will try to submit a proposal for the same.

@vamshin can you clarify that you are thinking this would not send metrics to anywhere for customers of open source opensearch, but would allow customers (managed service providers, customers, etc.) to send their own cluster data to their self-designated endpoint?

This can probably be covered with an addition to https://github.com/opensearch-project/OpenSearch/pull/10724
One thing to note is that we're trying to move away from stats APIs for everything (at least some of us are pushing in that direction). Instead, we're trying to embrace more of a tracing/telemetry-based approach, which should give us much better numbers, rather than silly point-in-time "how is the cluster right now?" data.

https://github.com/opensearch-project/OpenSearch/pull/10724
@deshsidd Is there a way I can use this work to get plugins query clause count?
My use case is
I need to cater neural search query clause count. For that we have neural query builder in neural search plugin. Now the problem is I cannot access the neuralquerybuilder in opensearch and therefore add a counter in SearchQueryCounter and SearchQueryCategorizingVisitor.

#10724 @deshsidd Is there a way I can use this work to get plugins query clause count?
My use case is I need to cater neural search query clause count. For that we have neural query builder in neural search plugin. Now the problem is I cannot access the neuralquerybuilder in opensearch and therefore add a counter in SearchQueryCounter and SearchQueryCategorizingVisitor.
cc: @msfroh

The best bet would probably be to emit the metrics based on the QueryBuilder::getName() method. Unfortunately, right now, we have a fixed list of counters.
Can we make the counters dynamic instead? Like instead of having a bunch of counter fields, we could have a ConcurrentHashMap<String, Counter>, where we allocate the counters based on the query types that we see? @deshsidd, what do you think?

@msfroh Yes that would make sense and will cover all the query types. Let me look into adding something like this in 2.13. Will add a task for it. Thanks @vibrantvarun

@deshsidd  is this still planned for 2.13?

@msfroh Yes that would make sense and will cover all the query types. Let me look into adding something like this in 2.13. Will add a task for it. Thanks @vibrantvarun

Hi,
Can we make sure that plugins are able to use this feature to provide metrics for their query clauses. If the metrics cannot be provided by plugins then the feature won't be complete.
cc: @msfroh

This is not prioritized for 2.13 but is in the roadmap. As mentioned earlier we would have to make the counters dynamic where we allocate the counters based on the query types we see.

This is not prioritized for 2.13 but is in the roadmap. As mentioned earlier we would have to make the counters dynamic where we allocate the counters based on the query types we see.

This will make the feature half baked. :(

@msfroh can you point me to the interface that needs to be extended by plugins to use this feature?

@navneet1v I have already tried to extend it in plugins but the infrastructure is created like that it does not have feature to extend in plugin. Therefore, there is a need to do work at infrastructure level.
cc: @navneet1v @deshsidd

In theory, you shouldn't need to extend anything in a plugin. The change in https://github.com/opensearch-project/OpenSearch/pull/12601 just emits counters based on the query type.

@navneet1v @vibrantvarun As mentioned in the PR, the system now supports dynamic query types, eliminating the need to track specific query types exclusively. Instead, counters for all query types encountered including custom query types introduced by plugins will be created and incremented as needed. This ensures that accounting for any custom query types issued by plugins is also handled seamlessly.


fix(deps): update terraform terraform-ibm-modules/cbr/ibm to v1.29.0

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/cbr/ibm (source)
module
minor
1.28.1 -> 1.29.0

Release Notes

terraform-ibm-modules/terraform-ibm-cbr (terraform-ibm-modules/cbr/ibm)
v1.29.0
Compare Source
Features

added cloud logs support to the fscloud submodule profile (#568)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 3.19.7 :tada:
The release is available on:

GitHub release
v3.19.7

Your semantic-release bot :package::rocket:


Prevent ws client from publishing and receiving messages if the channel access is revoked

FEATURE REQUEST
It would be good to forcefully disconnect ws client, or otherwise prevent it from sending/receiving once the access to the channel has been revoked.
Currently, once the ws client is connected, no additional checks are done, so it can keep sending/receiving indefinitely.
I think this is a must have feature from security perspective.

This issue was probably resolved long ago, but let's test it before closing it.

@dborovcanin yes, tests confirm that this was resolved.


Setup project board for Intuitive and Accessible Documentation Editing

Jumping in to earn some hands on knowledge from the bb and hope the experience is what I need to take back what's mine


Docker root on volume

Fixes #73
Add facility so that one can ask d-s-s to prepare a seaparate volume for docker root directory.

Overall looks OK.  I actually didn't know about -g, I'd been just using symlinks myself.
Hmm, won't the admin doing this also have to relabel the target as var_t via e.g. semanage fcontext, or alternatively just mkdir -p && chcon -t var_t?  That'd be worth noting.

selinux labeling is a good point. I had been doing all my testing with selinux disabled (as for some reason selinux will not work with upstream docker). I will spend some time on figuring out this selinux bit.

Make sure you have docker-selinux installed and upstream should work fine.

@cgwalters  I have taken care of review comments and pushed patches again. PTAL.

@cgwalters ping.

I think this one should be run by the Fedora Docker team at least, not just me.
I'm a bit uncertain about automatically running semanage - that's a highly privileged operation as it persistently changes the policy.  Raw chcon --reference doesn't (but also isn't "persistent", though I think people running restorecon -R / etc. have more serious problems).
Some of this feels like it'd actually be better living in an installation setup program (i.e. Ansible).  Particularly for the complex cases where e.g. one wants to do "converged" storage in a Kubernetes cluster where a portion of the host's disk drives go to persistent volumes, a portion goes to caching images and containers, etc.
That said we do need to make the "single node" case easy, and particularly "switch single node to overlayfs on boot" would be nice.
With this we're starting to grow a lot of functionality and complexity without many test cases to match.  There is some work on that at my employer internally and I think we could try to extend coverage to make use of some of these options.

@rhatdan
Can you please look at this PR from selinux point of view.

@cgwalters, Your comment is combination of multiple issues, so I will break down my responses and respond to each issue separately.
Though we are changing policy in persistent manner using "semanage" but scope of policy change is limited. That is it happens only for a directory and happens only if that directory is created by d-s-s.
Similarly restorecon impact is limited to only container root and this newly created directory.
But I will let Dan Walsh decide if it is alright in the context of d-s-s or not.

Second core issue is that should this functionality be in d-s-s or not. I agree that d-s-s is getting complicated now but it is just offering more complicated storage setup now. I feel that we can have storage setup functionality in d-s-s as well as higher layers. And if installer chooses to setup storage, then they can just disable d-s-s by putting "STORAGE_DRIVER=" in /etc/sysconfig/docker-storage.
I am looking at it from pure fedora host perspective also. Where a user might just add another disk and might want to configure storage there. Or leave extra disk space and might want to configure storage.
My experience tells me that getting extra configuration screens in anaconda is hard. In fact even one gets it in, maintaining it as a separate package is much easier then when it runs in the context of installer.
So while I am not opposed to somebody introducing custom storage setup features in installers, but that should not preclude enhancing d-s-s. This does solve the case of single host nicely without worrying about what installer is being used.

Third issue is about automated test cases. I agree that being able to put automated test cases is very nice. You mentioned that you are already doing some work on automated test cases and these could be extended to test this. That makes sense. But I guess that's not a blocker for this patch series.

This one is now supported.


New version: PIKCLOUD.PikPak version 2.2.4

This pull request is automatically generated by 🥟 Dumplings in #3155
Microsoft Reviewers: Open in CodeFlow

@wingetbot run

Automatic Validation ended with:

2024-01-22T21:51:34.5328870Z Processing manifest: manifests\p\PIKCLOUD\PikPak\2.2.1
2024-01-22T21:51:34.5339636Z -----------------------------------------------------
2024-01-22T21:51:34.5346371Z Status: Started
2024-01-22T21:51:34.5354617Z Manifest file is Deleted
2024-01-22T21:51:34.7956887Z Status: Completed
2024-01-22T21:51:34.7966726Z ##[error] Result: Failed
2024-01-22T21:51:34.8021104Z ##[error] Sequence contains no elements
2024-01-22T21:51:34.8596753Z ##[error]Manifest Validation Failed

(Automated response - build 675.)

Sequence contains no elements

https://github.com/microsoft/winget-pkgs/issues/133371

(Automated response - build 675.)


Crashing on uncaught asynchronous exception

Given this gulp task:
gulp.task('test', function() {
    return gulp.src("*.spec.js")
        .pipe(jasmine());
});

and this spec:
describe("a test", function () {
    it("should work", function (done) {
        setTimeout(function () {
            throw "Exception";
            done();
        }, 1);
    });
});

the output will be:
[16:24:21] Starting 'test'...
Running 1 specs.

a test
C:\POC\test-gulp-jasmine\test.spec.js:4
            throw "Exception";
            ^
Exception

It would be nice to be able to catch the exception and return gracefully, but I guess this must be managed by jasmine directly and might not be possible at all.
Things that could be related:

a catchException option: https://github.com/jasmine/jasmine/blob/master/src/core/Env.js#L138
an issue related to exceptions in asynchronous code: https://github.com/jasmine/jasmine/issues/529

I think it must be possible, as Mocha handles and reports these asynchronous exceptions (didn't have a time to look into how it's done there).


__multi3 optimization

Feature
Implement an optimization pass that would eliminate the __multi3 function from WASM binary during JIT by replacing it with ISA specific (mainly for x86_64 and arm64) sequences, and then inline such sequences into callsites that would allow further optimizations
Benefit
A lot of code dealing with cryptography would benefit form faster full width u64 multiplications where such __multi3 arises
Implementation
If someone would give a few hints about where to start I'd try to implement it by myself
Alternatives
Not that I'm aware of. Patching into calling come native library function is a huge overhead for modern CPUs (4 cycles for x86_64 for e.g. mulx or mul), and while it would be faster most likely, it's still far from optimal case on a hot path
As an example a simple multiply-add-carry function like a*b + c + carry -> (high, low) that accumulates into u128 without overflows compiles down to the listing below, and it can be a good test subject (transformed from wasm into wat, may be not the best readable)
(module
  (type (;0;) (func (param i32 i64 i64 i64 i64)))
  (func $mac (type 0) (param i32 i64 i64 i64 i64)
    (local i32)
    global.get $__stack_pointer
    i32.const 16
    i32.sub
    local.tee 5
    global.set $__stack_pointer
    local.get 5
    local.get 2
    i64.const 0
    local.get 1
    i64.const 0
    call $__multi3
    local.get 0
    local.get 5
    i64.load
    local.tee 2
    local.get 3
    i64.add
    local.tee 3
    local.get 4
    i64.add
    local.tee 4
    i64.store
    local.get 0
    local.get 5
    i32.const 8
    i32.add
    i64.load
    local.get 3
    local.get 2
    i64.lt_u
    i64.extend_i32_u
    i64.add
    local.get 4
    local.get 3
    i64.lt_u
    i64.extend_i32_u
    i64.add
    i64.store offset=8
    local.get 5
    i32.const 16
    i32.add
    global.set $__stack_pointer
  )
  (func $__multi3 (type 0) (param i32 i64 i64 i64 i64)
    (local i64 i64 i64 i64 i64 i64)
    local.get 0
    local.get 3
    i64.const<PHONE_NUMBER>
    i64.and
    local.tee 5
    local.get 1
    i64.const<PHONE_NUMBER>
    i64.and
    local.tee 6
    i64.mul
    local.tee 7
    local.get 5
    local.get 1
    i64.const 32
    i64.shr_u
    local.tee 8
    i64.mul
    local.tee 9
    local.get 3
    i64.const 32
    i64.shr_u
    local.tee 10
    local.get 6
    i64.mul
    i64.add
    local.tee 5
    i64.const 32
    i64.shl
    i64.add
    local.tee 6
    i64.store
    local.get 0
    local.get 10
    local.get 8
    i64.mul
    local.get 5
    local.get 9
    i64.lt_u
    i64.extend_i32_u
    i64.const 32
    i64.shl
    local.get 5
    i64.const 32
    i64.shr_u
    i64.or
    i64.add
    local.get 6
    local.get 7
    i64.lt_u
    i64.extend_i32_u
    i64.add
    local.get 4
    local.get 1
    i64.mul
    local.get 3
    local.get 2
    i64.mul
    i64.add
    i64.add
    i64.store offset=8
  )
  (table (;0;) 1 1 funcref)
  (memory (;0;) 16)
  (global $__stack_pointer (mut i32) i32.const 1048576)
  (global (;1;) i32 i32.const 1048576)
  (global (;2;) i32 i32.const 1048576)
  (export "memory" (memory 0))
  (export "mac" (func $mac))
  (export "__data_end" (global 1))
  (export "__heap_base" (global 2))
)

Hi @shamatar -- thanks for raising this issue! I agree that the lack of "architecture-specific acceleration" for wide arithmetic lowered into Wasm bytecode is suboptimal and annoying.
On the other hand, somehow recognizing that a Wasm module's internal function happens to be the __multi3 function from the Wasm toolchain's runtime, and then actually replacing that function with another function, seems very likely to me to break the abstraction boundary (and hence security and portability properties) that the Wasm runtime provides. Either we check that it is exactly the __multi3 that we expect (same Wasm bytecode body), in which case this is a very brittle optimization (breaks with any little Wasm-toolchain implementation detail change), or we don't, in which case we might replace the code with an implementation that does something different (incorrectly). So, for certain, we would not implement logic that somehow recognizes __multi3 by name as special.
I think that there is a way around this though: we could pattern-match the actual operations and use the faster architecture-specific lowering when possible. In this specific case, a pattern could match whatever subgraphs correspond to the fast x64 instructions (mulx and friends?). I don't know enough about the state of the art of computer arithmetic implementations on x64 or aarch64 to suggest specific mappings.
The place to put such lowerings would be in the ISLE DSL (see e.g. cranelift/codegen/src/isa/x64/lower.isle); we'd be happy to help if you want to try to implement something here.

Hey @cfallin
Thank you for a detailed response. I was only mentioning __multi3 replacement by name as I have seen in one of the old threads an idea to call external library for it for a speedup. But if name matching is potentially too fragile (it may be safely-enough allowed under some feature flag if such flag would exist :) ) I'd try to implement matching-by-logic. __multi3 body looks like a standard u64 -> u32 and u32 * u32 -> u64 approach for simulation and hopefully should be catchable.
As for "state of the art" - the best one can get is adx + mulx combo for generic case I believe, but e.g. llvm never emits mulx, so I'd start with trying to optimize into add + mul for x86_64
I'm not use that a file you have mentioned is a right place as https://github.com/bytecodealliance/wasmtime/blob/b69fede72f6df81edbb0b3286b01e0a101a1758f/cranelift/codegen/src/isa/x64/lower.isle#L916 has a rule for full width multiplication lowering already, but the optimization first needs to catch __multi3, and emit just imul x y for it + inline into callsite. May be it also requires more than one pass, but the first step is to get to just imul x y

catch __multi3, and emit just imul x y for it + inline into callsite.

That's true, one way this could work in the future is to recognize the whole function and replace it with an imul.i128 (if that's what you mean by "catch `__multi3, and emit just imul"?) but that would I think hit the brittleness issues I mentioned.
Maybe the right way to start is to draw the dataflow graph of the bytecode you listed above, and find subgraphs that correspond to what adx and mulx do? If we can write a pattern that matches each of those, maybe we can get something at least slightly better than what we have today.
Eventually we will also have an inliner (it's really an orthogonal improvement) and when we do, it could combine with good-enough pattern matching on the function body here to produce an inlined sequence of instructions. In other words, getting to a tight sequence of instructions, and inlining that at the callsite, are two separate problems so let's solve each separately.
cc @abrown and @jlb6740 for thoughts on the x86-64 lowering here...

to recognize the whole function and replace it - yes, it's a final goal, and ideally function body will be patter-matched somehow (and not by name). My impression on lower.isle file is that it's a definition of "simple" rules to lower some short sequences of CLIF instructions to machine instructions, and not the rules for "large" pattern matching.
I'll make a dataflow, just need some time

so it should be possible to do a lot better here, and without recent extensions like mulx/adx.

Yes, it's kind of possible due to two carry chains (in my example of u64 * u64 + u64 + u64), but it would be the next step (and adx is much more useful for even "larger" math, like u256 full width multiplication).
So my example's "optimal" code is like
mac:
        mov     r8, rdx
        mov     rax, rsi
        mul     rdi
        add     rax, r8
        adc     rdx, 0
        add     rax, rcx
        adc     rdx, 0
        ret

where adx will allow to speedup two carry propagation chains. mulx allows more flexible register allocation, but none of the compilers uses it in practice as far as I known

I've tried to insert an optimization into cranelift directly. I should be able to match the full body of the generated __multi3 and transform it into i128 multiplication. Please check the #4106

Made an initial functional PR, folds from
block0(v0: i64, v1: i64, v2: i32, v3: i64, v4: i64, v5: i64, v6: i64):
@5130                               v7 = iconst.i64 0
@5136                               v8 = iconst.i64 0xffff_ffff
@513c                               v9 = band_imm v5, 0xffff_ffff
@5141                               v10 = iconst.i64 0xffff_ffff
@5147                               v11 = band_imm v3, 0xffff_ffff
@514a                               v12 = imul v9, v11
@5151                               v13 = iconst.i64 32
@5153                               v14 = ushr_imm v3, 32
@5156                               v15 = imul v9, v14
@515b                               v16 = iconst.i64 32
@515d                               v17 = ushr_imm v5, 32
@5162                               v18 = imul v17, v11
@5163                               v19 = iadd v15, v18
@5166                               v20 = iconst.i64 32
@5168                               v21 = ishl_imm v19, 32
@5169                               v22 = iadd v12, v21
@516c                               v23 = heap_addr.i64 heap0, v2, 1
@516c                               store little v22, v23
@5175                               v24 = imul v17, v14
@517a                               v25 = icmp ult v19, v15
@517a                               v26 = bint.i32 v25
@517b                               v27 = uextend.i64 v26
@517c                               v28 = iconst.i64 32
@517e                               v29 = ishl_imm v27, 32
@5181                               v30 = iconst.i64 32
@5183                               v31 = ushr_imm v19, 32
@5184                               v32 = bor v29, v31
@5185                               v33 = iadd v24, v32
@518a                               v34 = icmp ult v22, v12
@518a                               v35 = bint.i32 v34
@518b                               v36 = uextend.i64 v35
@518c                               v37 = iadd v33, v36
@5191                               v38 = imul v6, v3
@5196                               v39 = imul v5, v4
@5197                               v40 = iadd v38, v39
@5198                               v41 = iadd v37, v40
@5199                               v42 = heap_addr.i64 heap0, v2, 1
@5199                               store little v41, v42+8
@519c                               jump block1

                                block1:
@519c                               return

into
block0(v0: i64, v1: i64, v2: i32, v3: i64, v4: i64, v5: i64, v6: i64):
                                    v43 = iconcat v3, v4
                                    v44 = iconcat v5, v6
                                    v45 = imul v43, v44
                                    v46, v47 = isplit v45
@5130                               v7 = iconst.i64 0
@5136                               v8 = iconst.i64 0xffff_ffff
@513c                               v9 = nop
@5141                               v10 = iconst.i64 0xffff_ffff
@5147                               v11 = nop
@514a                               v12 = nop
@5151                               v13 = iconst.i64 32
@5153                               v14 = nop
@5156                               v15 = nop
@515b                               v16 = iconst.i64 32
@515d                               v17 = nop
@5162                               v18 = nop
@5163                               v19 = nop
@5166                               v20 = iconst.i64 32
@5168                               v21 = nop
@5169                               v22 = nop
@516c                               v23 = heap_addr.i64 heap0, v2, 1
@516c                               store little v46, v23
@5175                               v24 = nop
@517a                               v25 = nop
@517a                               v26 = nop
@517b                               v27 = nop
@517c                               v28 = iconst.i64 32
@517e                               v29 = nop
@5181                               v30 = iconst.i64 32
@5183                               v31 = nop
@5184                               v32 = nop
@5185                               v33 = nop
@518a                               v34 = nop
@518a                               v35 = nop
@518b                               v36 = nop
@518c                               v37 = nop
@5191                               v38 = nop
@5196                               v39 = nop
@5197                               v40 = nop
@5198                               v41 = nop
@5199                               v42 = heap_addr.i64 heap0, v2, 1
@5199                               store little v47, v42+8
@519c                               jump block1

                                block1:
@519c                               return

Few comments:

is there any pass that removes unused values? I believe it should be, but I'd like to trigger it manually
it may be a better options if instead of concat - imul - split I'd try manual implementation using imul/umulhi
is there any inlining step? In my benchmarks I've tried when the main MAC arithmetic function is "extern C" and normal Rust with "inline" attribute and there is no difference, that most likely indicated that "__multi3" is inserted by the compiler backend at the later stages. And after I substantially reduce the body of "__multi3" it should be a good candidate for inlining, and it will also give me more information to work with, like the fact that parts of the input are always 0

is there any pass that removes unused values? I believe it should be, but I'd like to trigger it manually

Yes, the dce pass. Note that lowering to the backend specific ir already does this implicitly. Also I think you shouldn't replace the instructions with nops. It results in invalid clif ir (as nop doesn't return any values) and it is incorrect if any of the instruction return values are used elsewhere. For example because you matched a function that looks somewhat like __multi3 but not exactly.

is there any inlining step?

No, there isn't. It is also non-trivial to implement as currently every function is independently lowered to clif ir, optimized and codegened to native code. Wasmtime even compiles multiple functions in parallel.

Ty @bjorn3
I've analyzed my NOPing approach and it's overzealous indeed.
As for inlining - are there any plans to add it? It would require some form of "synchronization point" after CLIF generation, then inlining itself can also be parallelized, and then it's again a parallel native codegen again

Yes, https://github.com/bytecodealliance/rfcs/blob/main/accepted/cranelift-roadmap-2022.md#inlining

I've been investigating what we can do to improve the performance of __multi3. For reference, the C source for this function is in LLVM at compiler-rt/lib/builtins/multi3.c, which helps a little with understanding what's going on.
Part of the solution proposed in earlier discussion here is function inlining. @elliottt and I discussed this some yesterday and what I'd like to see is a separate tool that supports an inlining transformation on core WebAssembly. I'd then recommend that people use that to eliminate the optimization barriers around __multi3. I don't see inlining actually happening soon in Wasmtime itself and it'd be nice to have something to recommend for that.
One difficult problem is that this function returns a two-member struct, which according to the WebAssembly basic C ABI means it must be returned through linear memory. If it were returned on the wasm stack we'd be able to keep both parts in registers but instead we have to write to RAM. There is no optimization we can legally do to avoid the writes, even with inlining, although with inlining we likely can avoid reading the values back from RAM. Ideally the C ABI would be updated to take advantage of multi-value returns, which apparently didn't exist when it was specified.
Now, what is this function actually computing, and what patterns can we recognize and transform in the mid-end?

Raw CLIF for __multi3, slightly edited
test optimize precise-output
set opt_level=speed_and_size
target x86_64

; v3  = a_lo
; v4  = a_hi
; v5  = b_lo
; v6  = b_hi
; v11 = a0
; v14 = a1
; v9  = b0
; v17 = b1
function %multi3(i64 vmctx, i64, i32, i64, i64, i64, i64) fast {
    gv0 = vmctx
    gv1 = load.i64 notrap aligned readonly gv0+80

block0(v0: i64, v1: i64, v2: i32, v3: i64, v4: i64, v5: i64, v6: i64):
    v7 = iconst.i64 0
    v8 = iconst.i64 0xffff_ffff
    v9 = band v5, v8
    v10 = iconst.i64 0xffff_ffff
    v11 = band v3, v10
    v12 = imul v9, v11
    v13 = iconst.i64 32
    v14 = ushr v3, v13
    v15 = imul v9, v14
    v16 = iconst.i64 32
    v17 = ushr v5, v16
    v18 = imul v17, v11
    v19 = iadd v15, v18
    v20 = iconst.i64 32
    v21 = ishl v19, v20
    v22 = iadd v12, v21
    v23 = uextend.i64 v2
    v24 = global_value.i64 gv1
    v25 = iadd v24, v23
    store little heap v22, v25
    v26 = imul v17, v14
    v27 = icmp ult v19, v15
    v28 = uextend.i32 v27
    v29 = uextend.i64 v28
    v30 = iconst.i64 32
    v31 = ishl v29, v30
    v32 = iconst.i64 32
    v33 = ushr v19, v32
    v34 = bor v31, v33
    v35 = iadd v26, v34
    v36 = icmp ult v22, v12
    v37 = uextend.i32 v36
    v38 = uextend.i64 v37
    v39 = iadd v35, v38
    v40 = imul v6, v3
    v41 = imul v5, v4
    v42 = iadd v40, v41
    v43 = iadd v39, v42
    v44 = uextend.i64 v2
    v45 = global_value.i64 gv1
    v46 = iadd v45, v44
    v47 = iadd_imm v46, 8
    store little heap v43, v47
    return
}

This function implements a 128-bit multiply where the operands and result are stored as pairs of 64-bit integers. It builds up the result, in part, using a series of 32x32->64 multiplies. Viewed as a sequence of 32-bit "digits", the result of long multiplication should look like this, although each single-digit product may have a carry into the column to its left:
     a3    a2    a1    a0
   * b3    b2    b1    b0
  =======================
  a0*b3 a0*b2 a0*b1 a0*b0
+ a1*b2 a1*b1 a1*b0
+ a2*b1 a2*b0
+ a3*b0

However only a1/a0 and b1/b0 are actually treated this way, to compute the lower half of the result along with part of the upper half.
                 a1    a0
   *             b1    b0
  =======================
              a0*b1 a0*b0
+       a1*b1 a1*b0

In Cranelift, we'd want to implement this with a pair of instructions: imul and umulhi, to produce the lower and upper 64 bits, respectively.
The remaining part of the upper half is just a_lo*b_hi+a_hi*b_lo, performed as regular 64-bit multiplies which are equivalent to these parts of the 32-bit-at-a-time long multiplication:
  a0*b3 a0*b2
+ a1*b2
+ a2*b1 a2*b0
+ a3*b0

There's actually nothing we can improve in this part, as shown in @cfallin's gcc-generated assembly listing, which has two imulq and two addq.
So back to the lower-half 64x64->128 multiply, performed 32 bits at a time. Our current optimizer produces this sequence which is equivalent to v22 = imul.i64 v3, v5:
    v8 = iconst.i64 0xffff_ffff
    v9 = band v5, v8  ; v8 = 0xffff_ffff
    v11 = band v3, v8  ; v8 = 0xffff_ffff
    v12 = imul v9, v11
    v13 = iconst.i64 32
    v14 = ushr v3, v13  ; v13 = 32
    v15 = imul v9, v14
    v17 = ushr v5, v13  ; v13 = 32
    v18 = imul v17, v11
    v19 = iadd v15, v18
    v21 = ishl v19, v13  ; v13 = 32
    v22 = iadd v12, v21

Then I think this part (which reuses some of the intermediate values above) is equivalent to either umulhi or smulhi. I haven't yet figured out what purpose the icmp instructions serve, although I can see they're effectively checking if the adds overflowed for v22 and v19 above.
    v26 = imul v17, v14
    v27 = icmp ult v19, v15
    v49 = uextend.i64 v27
    v31 = ishl v49, v13  ; v13 = 32
    v33 = ushr v19, v13  ; v13 = 32
    v34 = bor v31, v33
    v35 = iadd v26, v34
    v36 = icmp ult v22, v12
    v51 = uextend.i64 v36
    v39 = iadd v35, v51

If we can get egraph rules to simplify these two sequences to the upper and lower halves of a 64-bit multiply, then once we also solve #5623 we'll get the same sequence of instructions out that gcc produces for 128-bit multiplies.
Alternatively, if we could match on multiple results at once then we could write a rule that matches this combination of three imul, one umulhi, and three iadd and turns them into a single 128-bit imul surrounded by iconcat and isplit. I believe on x64 we already lower that to something equivalent to the gcc-generated sequence.


[vultr] Handle network request errors

Closes #108

We're not reassigning anything to those variables. I'm curious to learn about the reason behind your suggestion. :)

no particular reason just worried when we use const with an array and object and we have to change the values needs more care even let and const have the same scope and also for some developers like me it becomes a little bit confusing how it is working fine when we are declaring an array with const and push and pop still working. and also this code will have public access i thought it should be better not let them confuse with non-reinitializing and just pushing and poping.
But if it is fine for you we can keep const.

I can totally understand your concern, but to maintain consistency with other commands, let's keep it as it. :)


Error handling

Rustlings - gives exercises with minor errors for us to fix and learn.
Originally the error I was getting was
the trait `From<CreationError>` is not implemented for `ParseIntError`

To fix this I tried:
impl From<CreationError> for ParseIntError{
    fn from(e: CreationError) -> ParseIntError{
        return ParseIntError::CreationError(e);
    }
}

impl From<ParseIntError> for CreationError{
	fn from(e: ParseIntError) -> CreationError{
		return CreationError::ParseIntError(e);
	}
}

This gave more errors.
Upon further reading, the error was fixed by
fn main() -> Result<(), Box<dyn Error>> {

But is there any way by which we can implement the trait From<CreationError>  for ParseIntError without boxing the error?

it has private fields and no public constructors


[SAC-250][CORE] Use hive models for remote HMS, rely on remote HMS for creating relevant entities

NOTE: We're in agreement for Use hive models for remote HMS, and but still not decided on rely on remote HMS for creating relevant entities. Even the decision happens like we should create Hive entities, hopefully it should be easy to bring the change based on this patch. So removing WIP and go for reviewing.
What changes were proposed in this pull request?
This patch proposes two major changes:

Use hive models (table, storagedesc, db) when Spark session enables Hive as well as remote HMS is being used (More clearer, hive.metastore.uris is set.) For all other cases like in-memory catalog or embedded HMS, SAC will use Spark models for table entity (and corresponding entities).

This also means SAC no longer relies on provider value of table to classify whether the table is hive or not. SAC has been in some weird situation - use Spark models but still follow Hive's qualifiedName rule when provider value for table is hive - SAC no longer does it, and only follow Hive's qualifiedName rule for Hive entities which will fit to Hive models.
Due to the change of classifying which is Hive table, the tests should be tested in both ways, without remote HMS and with remote HMS. beeju (Apache 2.0 License) is adopted to test with remote HMS - it launches embedded HMS and provides thrift interface, so we can set the value of hive.metastore.uris. Most of table related tests are now executed in both ways.

Only refer Hive table entity instead of creating Hive entities

This is based on the precondition that Hive Atlas hook in HMS is enabled. If Hive Atlas hook is enabled, Hive Atlas hook will handle all the DDL events and create entities in Atlas in prior to events SAC refers to the table in spark_process.
Given HMS is the "source of truth" for Hive tables/databases/etc, this is ideal to make Atlas Hive entities to be "up to date", as well as greatly reduce the size of request message on SAC.
Due to the change of Hive table as reference, this patch has to change basic interface as well as touch huge parts of codebase, since AtlasEntity is generally used for data structure across all the codebase. The new base data structure is AtlasReferenceable, which can represent both entity reference as well as entity with dependencies. Please refer the change of  AtlasEntityWithDependencies.scala file which contains changes of data structure. Most of code changes are done to conform with new data structures.
How was this patch tested?
Modified UTs, and existing ITs. Manually tested against non-remote HMS scenario -Apache Spark 2.4.0 & Apache Atlas 1.1.
(Remote HMS scenario is not easy to be set up locally, so would like to rely on UTs.)

table

query

spark.range(10).write.json("/tmp/sac/json-wip-entity-deps")
spark.sql("CREATE TABLE test_table_wip_entity_deps_a USING json location '/tmp/sac/json-wip-entity-deps'")
spark.sql("CREATE TABLE test_table_managed_wip_entity_deps_a SELECT * FROM test_table_wip_entity_deps_a")

screenshots

kind
sub-kind
screenshot

common
spark_process

common
spark_process - relationship

common
spark_db

common
spark_db - relationship

managed
spark_table

managed
spark_table - relationship

managed
spark_storagedesc

managed
spark_storagedesc - relationship

external
spark_table

external
spark_table - relationship

external
spark_storagedesc

external
spark_storagedesc - relationship

Closes #250

cc. @aadarshjajodia

Please note that this PR is manually tested with https://reviews.apache.org/r/70600/diff/3/ with slight modification (changed type of executionTime in spark_process to long).
Hoping that #249 is the last change on Spark model in near future, but honestly I can't promise. :) It would hold true unless we find bugs which has to modify models.

Assume you have tried with HMS? May be upload some screenshots with HMS if possible.

Unfortunately I don't have Hive installed locally, especially Hive hook should be installed and be functional as well. I'll try to get the chance to test manually.

Now the last blocker is setting up remote HMS in test environment and test it. :)

Finally I tested this against remote HMS and it worked as expected. :) Updated the PR description to add snapshots.

As the patch gets approval for every changes, I think we can check this in first and get post review if any. Merging this in. Thanks for reviewing!


Setup.py : zip_safe to permit static serve

During setup (apache + wsgi), I got errors about templates ("File not found").
Found this parameter to solve it. It prevents pip/easy_install to zip python module (used by egg installation mode) in installation directory.

Coverage increased (+0.6%) to 30.028% when pulling 5ef7c66377264c1736e7809068d8291023d15d53 on redref:zip_safe into 9fe0f091f3cfa4c1954d9bb2ac2e0aaf76710588 on voxpupuli:master.

I haven't run into this before so I had to do some reading.
http://peak.telecommunity.com/DevCenter/setuptools#setting-the-zip-safe-flag
If your building from the git tree you can build with setup.py easy_install --zip-ok to produce your zip file that will support the static serve without the modification for the source code.
@corey-hammerton, we never talked about if you were doing anything special to publish the python module.  I typically only release egg and whl.

I did not managed to catch clearly your opinion on this. But, to be clear, my problem came the other way. I would have needed setup.py easy_install --always-unzip to get it working.
Statics must certainly be served from an unzipped directory (not sure on templates). I think according to your link, this flag should be explicit.

I'm just trying to understand how we have released packages in the past.  I think this should be a good update.  Trying to understand why it works from pypi without the change.


Hive Metastore PostgreSQL compatibility in HiveTableLastUpdatedExtractor and PrestoViewMetadataExtractor

HiveTableLastUpdatedExtractor and PrestoViewMetadataExtractor now are only workable with MySQL connection string.
It will show syntax error when using Postgres.
The same kind of compatibility is already implemented in HiveTableMetadataExtractor. It should also be applied to other hive-related extractors.
Expected Behavior
There should be no problem using PostgreSQL. It should work same as MySQL.
Current Behavior
Traceback (most recent call last):
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
psycopg2.errors.SyntaxError: syntax error at or near "`"
LINE 2:     SELECT t.TBL_ID, d.NAME as `schema`, t.TBL_NAME name, t....
                                       ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trino.py", line 228, in <module>
    trino_view_loading_job.launch()
  File "/home/venv/lib/python3.9/site-packages/databuilder/job/job.py", line 76, in launch
    raise e
  File "/home/venv/lib/python3.9/site-packages/databuilder/job/job.py", line 64, in launch
    self._init()
  File "/home/venv/lib/python3.9/site-packages/databuilder/job/job.py", line 51, in _init
    self.task.init(self.conf)
  File "/home/venv/lib/python3.9/site-packages/databuilder/task/task.py", line 45, in init
    self.extractor.init(Scoped.get_scoped_conf(conf, self.extractor.get_scope()))
  File "/home/venv/lib/python3.9/site-packages/databuilder/extractor/presto_view_metadata_extractor.py", line 57, in init
    self._alchemy_extractor = sql_alchemy_extractor.from_surrounding_config(conf, self.sql_stmt)
  File "/home/venv/lib/python3.9/site-packages/databuilder/extractor/sql_alchemy_extractor.py", line 102, in from_surrounding_config
    ae.init(c)
  File "/home/venv/lib/python3.9/site-packages/databuilder/extractor/sql_alchemy_extractor.py", line 40, in init
    self._execute_query()
  File "/home/venv/lib/python3.9/site-packages/databuilder/extractor/sql_alchemy_extractor.py", line 65, in _execute_query
    self.results = self.connection.execute(self.extract_sql)
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1003, in execute
    return self._execute_text(object_, multiparams, params)
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1172, in _execute_text
    ret = self._execute_context(
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
    self._handle_dbapi_exception(
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
    util.raise_(
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
    raise exception
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
    self.dialect.do_execute(
  File "/home/venv/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.SyntaxError) syntax error at or near "`"
LINE 2:     SELECT t.TBL_ID, d.NAME as `schema`, t.TBL_NAME name, t....
                                       ^

[SQL:
    SELECT t.TBL_ID, d.NAME as `schema`, t.TBL_NAME name, t.TBL_TYPE, t.VIEW_ORIGINAL_TEXT as view_original_text
    FROM TBLS t
    JOIN DBS d ON t.DB_ID = d.DB_ID
    WHERE t.VIEW_EXPANDED_TEXT = '/* Presto View */'

    ORDER BY t.TBL_ID desc;
    ]
(Background on this error at: http://sqlalche.me/e/13/f405)

Possible Solution
I have already written equivalent PostgreSQL statement for those syntax errors. Maybe I could make it both compatible following the approach in MetadataExtractor and submit a PR later.
HiveTableLastUpdatedExtractor
The commented out part is the original MySQL
# PARTITION_TABLE_SQL_STATEMENT = """
# SELECT
# DBS.NAME as `schema`,
# TBL_NAME as table_name,
# MAX(PARTITIONS.CREATE_TIME) as last_updated_time
# FROM TBLS
# JOIN DBS ON TBLS.DB_ID = DBS.DB_ID
# JOIN PARTITIONS ON TBLS.TBL_ID = PARTITIONS.TBL_ID
# {where_clause_suffix}
# GROUP BY `schema`, table_name
# ORDER BY `schema`, table_name;
# """

# Postgresql
PARTITION_TABLE_SQL_STATEMENT = """
SELECT
d."NAME" as "schema",
t."TBL_NAME" as table_name,
MAX(p."CREATE_TIME") as last_updated_time
FROM "TBLS" t
JOIN "DBS" d ON t."DB_ID" = d."DB_ID"
JOIN "PARTITIONS" p ON t."TBL_ID" = p."TBL_ID"
{where_clause_suffix}
GROUP BY "schema", table_name
ORDER BY "schema", table_name;
"""

# NON_PARTITIONED_TABLE_SQL_STATEMENT = """
# SELECT
# DBS.NAME as `schema`,
# TBL_NAME as table_name,
# SDS.LOCATION as location
# FROM TBLS
# JOIN DBS ON TBLS.DB_ID = DBS.DB_ID
# JOIN SDS ON TBLS.SD_ID = SDS.SD_ID
# {where_clause_suffix}
# ORDER BY `schema`, table_name;
# """

# Postgresql
NON_PARTITIONED_TABLE_SQL_STATEMENT = """
SELECT
d."NAME" as "schema",
t."TBL_NAME" as table_name,
s."LOCATION" as location
FROM "TBLS" t
JOIN "DBS" d ON t."DB_ID" = d."DB_ID"
JOIN "SDS" s ON t."SD_ID" = s."SD_ID"
{where_clause_suffix}
ORDER BY "schema", table_name;
"""

PrestoView
# SQL_STATEMENT = """
# SELECT t.TBL_ID, d.NAME as `schema`, t.TBL_NAME name, t.TBL_TYPE, t.VIEW_ORIGINAL_TEXT as view_original_text
# FROM TBLS t
# JOIN DBS d ON t.DB_ID = d.DB_ID
# WHERE t.VIEW_EXPANDED_TEXT = '/* Presto View */'
# {where_clause_suffix}
# ORDER BY t.TBL_ID desc;
# """

# PostgreSQL
SQL_STATEMENT = """
SELECT t."TBL_ID",
d."NAME" as "schema",
t."TBL_NAME" as name,
t."TBL_TYPE",
t."VIEW_ORIGINAL_TEXT" as view_original_text
FROM "TBLS" t
JOIN "DBS" d ON t."DB_ID" = d."DB_ID"
WHERE t."VIEW_EXPANDED_TEXT" = '/* Presto View */'
{where_clause_suffix}
ORDER BY t."TBL_ID" desc;
"""

Steps to Reproduce
Provide a PostgreSQL connection string to the extractor, it will fail and show the syntax error.
job_config = ConfigFactory.from_dict({
        f'extractor.presto_view_metadata.extractor.sqlalchemy.{PrestoViewMetadataExtractor.WHERE_CLAUSE_SUFFIX_KEY}': where_clause_suffix,
        f'extractor.presto_view_metadata.extractor.sqlalchemy.{SQLAlchemyExtractor.CONN_STRING}': metastore_postgresql_connection_string(),
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.NODE_DIR_PATH}': node_files_folder,
        f'loader.filesystem_csv_neo4j.{FsNeo4jCSVLoader.RELATION_DIR_PATH}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NODE_FILES_DIR}': node_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.RELATION_FILES_DIR}': relationship_files_folder,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_END_POINT_KEY}': neo4j_endpoint,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_USER}': neo4j_user,
        f'publisher.neo4j.{neo4j_csv_publisher.NEO4J_PASSWORD}': neo4j_password,
        f'publisher.neo4j.{neo4j_csv_publisher.JOB_PUBLISH_TAG}': timestamp,  # give the timestamp to each node in neo4j
    })
    job = DefaultJob(conf=job_config,
                     task=DefaultTask(extractor=PrestoViewMetadataExtractor(), loader=FsNeo4jCSVLoader()),
                     publisher=Neo4jCsvPublisher())

Bump, PR for the fix is on the way!


Implement uninstall/reinstall interface for argocd plugin

Description
At the moment, the Reinstall and Uninstall are mocks.
We want to implement them.
Describe the Proposal
Reinstall: helm upgrade --install
Uninstall: helm delete

/assign @daniel-hutao


DOC: some chat_id can't found

Location of the documentation
[this should provide the location of the documentation, e.g. "CONTRIBUTION.md" or the URL of the documentation, e.g. "https://github.com/tangyoha/telegram_media_downloader/blob/master/CONTRIBUTING.md"]
Documentation problem
[this should provide a description of what documentation you believe needs to be fixed/improved]
Suggested fix for documentation
[this should explain the suggested fix and why it's better than the existing documentation]

什么意思


Revert 8fabb5c3ef8b928

Fixes https://github.com/divio/django-cms/issues/5907
This reverts commit 8fabb5c3ef8b9283b535ea1b76f029c51d3e9c29.
The commit in question introduced a regression.

Coverage remained the same at 82.604% when pulling f790f7549a77ef71d121e0d4a45cfca8b129465e on czpython:revert/3.4.x/8fabb5c3ef8b928 into c330dae6db7bc5585d968126375ca0c5cb58b417 on divio:release/3.4.x.


Refactor transaction builder to build up delayed computation

Instead of taking the rng parameter for each step in the transaction builder, we can take it once, in the finalize step. Internally, this would look like building up a "script" of actions to perform using the rng, and executing that script upon finalization.

We could also take an owned RNG at the beginning of the transaction builder and save it internally. This way, we could evaluate the actions eagerly rather than lazily, and avoid having to build an internal script.

Closing this for now since the design has shifted (and will shift further), we could re-evaluate after finishing wallet refactoring.


WIP: Add ZOL program for confidential transactions

Mostly scaffolding. Not much to see here yet.

Moving to back burner


Incorporate FreeBSD's local changes (2 files)

Just the last commit (Aug 24).

Most of this is already in 0.4.5;  The “.Dq” issue is that the argument needs quoted (.Dq “Do the…”), but I don’t see the need for an initializer for “code”, since it’s only referenced inside the for loop.  Is this addressing a bug, a compiler warning or just defensive coding?
Thanks,
Phil

On Aug 24, 2015, at 12:44 PM, xcllnt<EMAIL_ADDRESS>wrote:
Just the last commit (Aug 24).
You can view, comment on, or merge this pull request online at:
https://github.com/Juniper/libxo/pull/48 https://github.com/Juniper/libxo/pull/48
Commit Summary
s/elseif/elif/
Fix synopsis: s/container/instance/g
Remove stray THREAD_UNLOCK() from xo_send_syslog()
Match argument to format specifier for ILP32 archirectures.
Merge remote-tracking branch 'upstream/master'
Merge branch 'master' of https://github.com/xcllnt/libxo
Merge remote-tracking branch 'upstream/master'
Merge remote-tracking branch 'upstream/master'
FreeBSD's local changes:
File Changes
M libxo/libxo.c https://github.com/Juniper/libxo/pull/48/files#diff-0 (2)
M libxo/xo_open_container.3 https://github.com/Juniper/libxo/pull/48/files#diff-1 (4)
Patch Links:
https://github.com/Juniper/libxo/pull/48.patch https://github.com/Juniper/libxo/pull/48.patch
https://github.com/Juniper/libxo/pull/48.diff https://github.com/Juniper/libxo/pull/48.diff
—
Reply to this email directly or view it on GitHub https://github.com/Juniper/libxo/pull/48.

On Aug 24, 2015, at 10:46 AM, Phil Shafer<EMAIL_ADDRESS>wrote:
Most of this is already in 0.4.5; The “.Dq” issue is that the argument needs quoted (.Dq “Do the…”), but I don’t see the need for an initializer for “code”, since it’s only referenced inside the for loop. Is this addressing a bug, a compiler warning or just defensive coding?

With -Werror, the compiler warning breaks the build. It’s both a bug (GCC’s bug)
and a compiler warning. The assignment is a work-around.
BTW: I just committed 0.4.5 and got a report that the build is broken for ARM.
I'm testing the following fix:
Index: contrib/libxo/libxo/xo_wcwidth.h
--- contrib/libxo/libxo/xo_wcwidth.h	(revision 287111)
+++ contrib/libxo/libxo/xo_wcwidth.h	(working copy)
@@ -62,8 +62,8 @@
#include <wchar.h>
struct interval {

int first;
int last;

wchar_t first;
wchar_t last;
};

/* auxiliary function for binary search in interval table */
The problem is that wchar_t is defined as signed int for some architectures
and unsigned int for others. With the compiler warning about comparisons
between different signedness and -Werror, using just int is problematic.
I’ll create a pull request (without hopefully all the previous changes) when I
have a final fix.
FYI,
--
Marcel Moolenaar
<EMAIL_ADDRESS><EMAIL_ADDRESS>


wikipedia: increase REXML entity expansion limit during XML parsing

Using Datasets::Wikipedia#each raised an entity expansion has grown too large (RuntimeError). This error occurs because the entity expansion limit in REXML is set by https://github.com/ruby/rexml/pull/187, and Datasets::Wikipedia#each exceeds that limit.
In Red Datasets, increasing the entity expansion limit is not a problem because we want to handle large datasets. Therefore, we temporarily increase the limit.
require 'datasets'

wikipedia = Datasets::Wikipedia.new
wikipedia.each do |wiki|
  pp wiki
end

$ cd red-datasets && bundle && bundle exec ruby wiki
/home/otegami/.rbenv/versions/3.3.3/lib/ruby/gems/3.3.0/gems/rexml-3.3.4/lib/rexml/parsers/baseparser.rb:560:in `block in unnormalize': entity expansion has grown too large (RuntimeError)

entity expansion has grown too large (RuntimeError) might be resolved at stream.

https://github.com/ruby/rexml/issues/193

After this, we might face the following exception.
I will make sure whether the original data is problem or not.
$ cat example/wikipedia.rb
#!/usr/bin/env ruby

require "datasets"

wikipedia = Datasets::Wikipedia.new(language: :en, type: :articles)
wikipedia.each do |page|
  p [
    page.title,
    page.namespace,
    page.id,
    page.restrictions,
    page.redirect,
    page.revision
  ]
end
$ ruby example/wikipedia.rb
Failed to read bzcat input: Errno::EPIPE: Broken pipe
/home/kodama/.rbenv/versions/3.3.2/lib/ruby/gems/3.3.0/gems/rexml-3.3.4/lib/rexml/parsers/streamparser.rb:25:in `parse': Missing end tag for '/mediawiki/page/revision/text' (REXML::ParseException)
Line: -1
Position: -1
Last 80 unconsumed characters:

	from /home/kodama/work/ruby/red-datasets/lib/datasets/wikipedia.rb:52:in `block (2 levels) in each'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/wikipedia.rb:91:in `with_increased_entity_expansion_text_limit'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/wikipedia.rb:51:in `block in each'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/dataset.rb:78:in `block (2 levels) in extract_bz2'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `pipe'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `block in extract_bz2'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `pipe'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `extract_bz2'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/wikipedia.rb:76:in `open_data'
	from /home/kodama/work/ruby/red-datasets/lib/datasets/wikipedia.rb:48:in `each'
	from example/wikipedia.rb:6:in `<main>'

It seems that your download failed.
Could you retry with wikipedia.clear_cache! before wikipedia.each?

I'm sorry for the delay in checking.
The following error has occurred, and it seems necessary to determine if it's a data error on the Wikipedia side or if the download didn't go well.
However, my home internet is slow, and it might take some time to check.
If you don't mind, I would appreciate it if you could help me check it together.
bundle exec ruby example/wikipedia.rb
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/3.3.0/json/common.rb:3: warning: ostruct was loaded from the standard library, but will no longer be part of the default gems starting from Ruby 3.5.0.
You can add ostruct to your Gemfile or gemspec to silence this warning.
enwiki-latest-pages-articles.xml.bz2 - 075.2% [ 17.28GB/ 22.97GB] 04:11:52  47KB/sB/ss
bzcat: Data integrity error when decompressing.
	Input file = (stdin), output file = (stdout)

It is possible that the compressed file(s) have become corrupted.
You can use the -tvv option to test integrity of such files.

You can use the `bzip2recover' program to attempt to recover
data from undamaged sections of corrupted files.

Failed to read bzcat input: Errno::EPIPE: Broken pipe
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:528:in `rescue in pull_event': #<ArgumentError: invalid byte sequence in UTF-8> (REXML::ParseException)
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/source.rb:231:in `scan_until'
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/source.rb:231:in `read_until'
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:507:in `pull_event'
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:242:in `pull'
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/streamparser.rb:32:in `parse'
/home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:52:in `block (2 levels) in each'
/home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:91:in `with_increased_entity_expansion_text_limit'
/home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:51:in `block in each'
/home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:78:in `block (2 levels) in extract_bz2'
/home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `pipe'
/home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `block in extract_bz2'
/home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `pipe'
/home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `extract_bz2'
/home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:76:in `open_data'
/home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:48:in `each'
example/wikipedia.rb:5:in `<main>'
...
Exception parsing
Line: -1
Position: -1
Last 80 unconsumed characters:
{{short description|American rapper}} {{Use mdy dates|date=February 2021}} {{UDP|
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:432:in `pull_event'
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:242:in `pull'
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/streamparser.rb:32:in `parse'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:52:in `block (2 levels) in each'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:91:in `with_increased_entity_expansion_text_limit'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:51:in `block in each'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:78:in `block (2 levels) in extract_bz2'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `pipe'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `block in extract_bz2'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `pipe'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `extract_bz2'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:76:in `open_data'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:48:in `each'
	from example/wikipedia.rb:5:in `<main>'
/home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/source.rb:231:in `scan_until': invalid byte sequence in UTF-8 (ArgumentError)

      until str = @scanner.scan_until(pattern)
                                      ^^^^^^^
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/source.rb:231:in `read_until'
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:507:in `pull_event'
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/baseparser.rb:242:in `pull'
	from /home/otegami/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/rexml-3.3.8/lib/rexml/parsers/streamparser.rb:32:in `parse'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:52:in `block (2 levels) in each'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:91:in `with_increased_entity_expansion_text_limit'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:51:in `block in each'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:78:in `block (2 levels) in extract_bz2'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `pipe'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:56:in `block in extract_bz2'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `pipe'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/dataset.rb:55:in `extract_bz2'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:76:in `open_data'
	from /home/otegami/work/ruby/red-datasets/lib/datasets/wikipedia.rb:48:in `each'
	from example/wikipedia.rb:5:in `<main>'

Could you retry it to check whether the error is reproducible?


ff:tooltip causes Vue.js issues near logic statements

Current Behavior
The tooltip directive introduced in https://github.com/flowforge/forge-ui-components/pull/53, seems to cause Vue.js rendering troubles sometimes when added to DOM elements containing Vue.js render logic.
In https://github.com/flowforge/flowforge/pull/1311, ff:tooltip was removed from the wrapping span, as during the **re-**rendering of the two sub-elements Vue.js was throwing errors related to being unable to find the parentNode or insertBefore an element. I believe this is due to ff:tooltip using .innerHTML to modify the DOM, which is likely throwing Vue.js's state out of sync.
https://github.com/flowforge/forge-ui-components/blob/021fbad67445d61d58bfcbef795c2b8d52385181/src/directives/Tooltip.js#L3-L13
Expected Behavior
ff-tooltip can append to the DOM without causing Vue.js exceptions
Steps To Reproduce
Wrap several conditionally rendered DOM elements with an element that has v-ff-tooltip set.
For example (pseudocode):

    On
    Off

Branch with the issue: https://github.com/flowforge/forge-ui-components/tree/demo-issue
Run:

npm serve
Head to http://localhost:8080/
Try to toggle the state and see the exceptions in the dev console

Environment

FlowForge version:
Node.js version:
npm version:
Platform/OS:
Browser:

I was able to reproduce this outside of forge-ui-components entirely: https://github.com/Pezmc/vuejs-append-to-innerHTML-issue. It revolves around modifying the DOM of an section of the page controlled by Vue.
Swapping to using Element.append() seems to sidestep the issue and Vue.js plays along nicely: https://github.com/Pezmc/vuejs-append-to-innerHTML-issue/compare/vuejs-append-to-dom-sidestep?expand=1


Bugfix - Dir - Windows: Paths not loaded correctly

Issue: @glennsl and I started experimenting with using dir and fp in Onivim 2, in this PR: https://github.com/onivim/oni2/pull/1703 - the API is perfect for us!
However, we hit a difficult-to-troubleshoot crash on startup when we integrated dir, and it wasn't clear what was going wrong - it was crashing before we even ran our CLI parsing or initialized our logging system.
It turned out, there were a couple of problems we hit:
First, there this was this exception:
(Invalid_argument "First token in path C:\\Users\\bryphe is not absolute.")
Raised at file "src/fp/Fp.re", line 220, characters 6-98
Called from file "src/dir/Dir.re", line 117, characters 16-35
Called from file "src/dir/Dir.re", line 187, characters 13-48
Called from file "src/dir/Dir.re", line 341, characters 8-20

This came from Fp, because it doesn't handle the windows back-slashes currently (this is documented limitation, but it's problematic with Dir - because it by default picks up windows-style paths for the directories, and breaks when it uses Fp to manipulate paths on windows).
I fixed this by normalizing the paths prior to sending to Fp (similar to what we do in esy) - however, the best longer-term fix would be to have Fp handle back-slashes too.
After fixing that, there was still a crash, and this time it was a native crash - not from OCaml. It turns out the sh_get_folder_path c stub wasn't actually returning the path string, so the runtime would crash natively when trying to process the Val_unit result as a string. I fixed this by returning a constructed option(string) from the API.
Lastly, I tweaked the API - because the directory construction methods have the possibility of throwing, I changed them to be deferred - it'll construct and give out the paths on request. This is preferable for use case, because we'd like to initialize our logging infra prior to anything that can throw, so if there is an error or exception, we can capture it and drop a crash log.

This looks great! Thanks for debugging. I'll let @jordwalke take a look before merging since he wrote most of this.

Thanks @kyldvs ! Just chatted with @jordwalke - I realized I need to update the dir test cases to pick up the new API. It's also curious why the native crash wasn't manifesting on Windows - I would've expected a crash in Dir.home, so I'll take a look at that too.

It's possible windows CI just isn't actually running things, for example our test cmd is a no-op in windows so it always passes, but it will run in bash environments:
https://github.com/facebookexperimental/reason-native/blob/master/scripts/test-ci.cmd

The test errors seem unrelated? It looks like refmterr tests behaving differently. Not sure how your changes could have caused that. I'll merge anyways if possible.

Looks like one of the test outputs has "because it is in the condition of an if-statement" in one of the error messages, which doesn't match what is expected, but that was likely added in a new ocaml compiler. I wonder if it's not using the lockfile to install / run tests.

I think that this is needed in the esy.json file to make the tests pass. We run them internally with that version and they pass, but the esy.json here doesn't have that. Want to try adding that and seeing if it makes them pass?
  "devDependencies": {
    "ocaml": "~4.7.1"
  }

Hey sorry for the slow reply @jordwalke ! Thanks for the tip re: 4.7.1 - I ran on OSX and almost everything passes, except for the refmterr test:
 FAIL  refmterr
  • refmterr › file_SyntaxError

    expect.string(received).toEqual(expected)

    Expected value to equal:
      "\027[27m\027[24m\027[2m\027[1m# Unformatted Error Output:\027[27m\027[24m\027[22m
      \027[2m# \027[22m\027[2mFile \"tests/__tests__/refmterr/__fixtures__/file_SyntaxError/file_SyntaxError_7.re\", line 20, characters 22-23:\027[22m
      \027[2m# \027[22m\027[2mError: Syntax error: ')' expected\027[22m
...
    Received:
      "\027[27m\027[24m\027[2m\027[1m# Unformatted Error Output:\027[27m\027[24m\027[22m
      \027[2m# \027[22m\027[2mFile \"tests/__tests__/refmterr/__fixtures__/file_SyntaxError/file_SyntaxError_7.re\", line 20, characters 22-23:\027[22m
      \027[2m# \027[22m\027[2mError: Unclosed \"{\" (opened line 10, column 4)\027[22m
...
    Difference:
      # Unformatted Error Output:
      # File "tests/__tests__/refmterr/__fixtures__/file_SyntaxError/file_SyntaxError_7.re", line 20, characters 22-23:
      # Error: Unclosed "{" (opened line 10, column 4)# Unformatted Error Output:
      # File "tests/__tests__/refmterr/__fixtures__/file_SyntaxError/file_SyntaxError_7.re", line 20, characters 22-23:
      # Error: Syntax error: ')' expected22m

I'll push up the change and the test command w/o the filter and we'll see what happens - maybe it's just an issue on my local machine 🤞

@bryphe I recall having issues with this refmterr test when updating things, it might just be a bad test that we can disable.

@bryphe I recall having issues with this refmterr test when updating things, it might just be a bad test that we can disable.

Thanks @kyldvs ! I disabled that test on Windows. Linux / macOS are green now.
Looks like, unfortunately, there are still failures on Windows, though (besides the Dir/Fp modules that were fixed in this PR).

I just applied the --filter Dir for the Windows test - so we can run the subset that were unblocked here. As a next step - we can start to unblock more and enable the full test suite


Error compiling in Windows.

BAS files with Windows CRLF will fail to compile. Converting them to LF and all works good.

Fixed. Many thanks for reporting!


Conflict with kind command related to Kubernetes

Describe the bug
There is a Kubernetes' tool called kind used to run local Kubernetes clusters using Docker. To execute it we use kind [command]. This conflicts with the kind command from kind-lang.
To Reproduce

Install kind (Kubernetes)

I used brew install kind

Use the command kind <some_file>.kind to type check a term on the base dir
It will display an error about using kind from Kubernetes: kind creates and manages local Kubernetes clusters using Docker container 'nodes'

Obs: to uninstall kind (Kubernetes): brew uninstall kind
Expected behavior
I expect to run kind-lang command's when inside Kind/base directory.
Environment (please complete the following information):

OS: macOS Big Sur 11.4
Kind version<EMAIL_ADDRESS>

Closed because the repository now only contains the code for the Kind2 language.


Fix the wrong comments.

Please answer these questions before submitting pull request

Why submit this pull request?

[ ] Bug fix

[ ] New feature provided

[ ] Improve performance

[x] comments

Related issues
As title said.

Codecov Report

Merging #4139 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4139      +/-   ##
==========================================
+ Coverage   27.17%   27.18%   +<.01%
==========================================
  Files        1141     1141
  Lines       25057    25057
  Branches     3625     3625
==========================================
+ Hits         6810     6812       +2
+ Misses      17644    17642       -2
  Partials      603      603

Impacted Files
Coverage Δ

...brary/client/jdbc/hikaricp/JDBCHikariCPClient.java
0% <ø> (ø)
:arrow_up:

...ache/skywalking/apm/agent/core/jvm/JVMService.java
77.04% <0%> (+3.27%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4aac329...d40f7f2. Read the comment docs.


feat: VoyageAI encoder

New Feature: Integration of super fast and accurate VoyageAI Embeddings into semantic-router

thanks @xRiddin — running again

@xRiddin you can run poetry lock --no-update to recreate the poetry.lock file — should fix the test issue above

@jamescalam done 👍

@jamescalam also the the maximum length of the list is 128 for Voyage AI api. If the user feeds an insane amount of data inside the lists, it will likely throw an error. Need to find a workaround for that.

Running:
poetry run pytest tests/unit/encoders/test_voyageai.py

Returns errors (same as being returned in the tests here), seem to only be mocking problems afaict — @xRiddin are you able to resolve? Let us know if you want us to jump in and help on the remaining mocking/tests
I also moved voyageai from a required install to an optional install, and modified the import in the encode file accordingly. Also renamed client to _client to align with other API-based encoders (for example the Mistral encoder)

@jamescalam I'm not able to resolve the mocking problems, any help on the remaining tests are very much appreciated.


Failed loading dll

Built a stub without the vmprotect option and upload it to virustotal. A while later, if you get a notification in your webhook, then it means the grabber is working but it ONLY not working in your PC. In simple term, the grabber will work if you send it to someone.

@Xarond12 Contact me on discord if you wanna discuss the issue. Blank?#0886

Please reply ASAP, or I'd have to mark this issue as stale.

This issue is related with pyinstaller and is device specific. Try on other device.


Yeasted Doughnuts | America's Test Kitchen Recipe

http://www.americastestkitchen.com/recipes/12601-yeasted-doughnuts

Yeasted Doughnuts _ America's Test Kitchen.pdf


Missing LICENSE file

What are your restrictions about using your code?

Hi, I added a MIT licence. Sorry for the long delay!


Fix acronym casing in webhook-event-delivery.md

http -> HTTP

@sixlettervariables : Thanks for your contribution! The author(s) have been notified to review your proposed change.


fix: type shadowing in proto file

When updating the dependencies in the client we started getting this error when generating the rust types from proto files:
Error:   × name 'account.AccountInfo' is shadowed
       ╭─[responses.proto:164:5]
   163 │     // Account info (with details for on-chain accounts)
   164 │     account.AccountInfo account = 1;
       ·     ─────────┬─────────
       ·              ╰── found here
   165 │ }
       ╰────
    help: 'account.AccountInfo' is is resolved to
          'responses.GetAccountDetailsResponse.account.AccountInfo',
          which is not defined. The innermost scope is searched first
          in name resolution. Consider using a leading '.'(i.e.,
          '.account.AccountInfo') to start from the outermost scope.

It seems the name of the field account is shadowing the name of the import account. As it is recommended in the message, this can be fixed by specifying the root.

The account name is used in two places in this context:

For the name of the first field GetAccountDetailsResponse
For the name of the module that is imported that contains the type AccountInfo

From what I understood, the first name is overwriting (also called shadowing) the second name in this case, so the type AccountInfo is being searched inside the field when it should be searched inside the module. By specifying the root with the '.' we are removing this ambiguity and the type will be searched in .account.AccountInfo.
This name collision doesn't happen in the other messages and that's why it's only needed here.

Yes, that would also work but the name should also be changed in the code in that case. I just looked into it and it's not that much different, maybe we prefer the name change instead of the explicit root.

I just looked into it and it's not that much different, maybe we prefer the name change instead of the explicit root.

I'm fine either way.

At the end I decided to change the name to keep the type definition consistent.


ci: prerender demo application for production

What's up with the ellipsis on the screenshot? 🤔

What's up with the ellipsis on the screenshot? 🤔

@waterplea
I think it is just flaky test.
The PR's changes dont affect any component, they are only about CI.


Bug-Service Discovery is not accepting dot characters

*Bug-Service Discovery is not accepting dot characters
Steps to Reproduce
Click on Service Discovery -> Add Record
enter dev.api.internal.test.net
dev. . sit. prod..
Click on Create
Add Record is not accepting . ( dot ) characters
Expected to add the DNS names as part of service discovery for aliases and for apis dns.
Result:
"Name" contains an invalid character: .
|Versions|Rancher v2.1.6 UI: v2.1.19 |
|Route|authenticated.project.dns.index|

Can you try to create what you want in k8s YAML and post it here, please?

Services create new DNS entries of the form <name>.<namespace>.svc.<cluster-domain>.  Names cannot create dots.  Services are not a mechanism for mapping arbitrary FQDNs to a value.  You can do that for a workload with additional hostfile entries (hostAliases).


Roll Plugins from a0b7daa5ec1f to 94e80fccb291 (2 revisions)

https://github.com/flutter/plugins/compare/a0b7daa5ec1f...94e80fccb291
2022-01-10<EMAIL_ADDRESS>[url_launcher] Fix minor memory leak in Linux url-launcher tests (flutter/plugins#4648)
2022-01-10<EMAIL_ADDRESS>Run tests only on main branch. (flutter/plugins#4645)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-plugins-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Allow servers to configure round end time

Create the console command dominatrix_roundendtime so that servers can configure the time which players are frozen while the winner is displayed on the screen. Currently hardcoded to 3 seconds.

Implemented. Will close ticket when released.


Remove ReactiveReadable and ReactiveWritable

ReactiveProperty is the only one that's useful so far.

🎊 My diff is ready for review at http://codereview.cc/D3029


this repo is dead?

First check https://github.com/SortableJS/Vue.Draggable/blob/master/CONTRIBUTING.md
Jsfiddle link
Step by step scenario
Actual Solution
Expected Solution

yes, I think it is... Sortable is no longer for vue 3 ;)

I tried Vue slicksort. But the repo has now been dormant since February. Its a neat Lib though: Vue Slicksort

Sortablejs-Vue3 is an almost drop-in replacement, that in my own experience works quire well: https://github.com/MaxLeiter/sortablejs-vue3


Bug: Editor lost its own selection when focus set inside external dialogs

Lexical version: 0.13.1
Steps To Reproduce

Type any text.
Click on 'Insert link' button.
Wait until the dialog opens and sets focus.
Click on 'Insert link' button a few times.

https://github.com/facebook/lexical/assets/3251933/4648471a-b677-4689-8db4-e359c82c361c
The current behavior
$getSelection() returns null.

Facing same problem, +1

I've seen users add onMouseDown={(e) => e.preventDefault()} to menu items to prevent losing editor focus/selection.
Obviously it would be nice if this wasn't an issue.


fix: making unmodifiablemap on getters

Contains
Problem in encapsulation in getters with lists/maps. Changed for unmodifiable versions.

Ignore this PR, made for a specific class in computer science college.


added CallbackStream

Hello.
This is a implementation of Stream that uses a callback as described in #70
Maybe should be great to add support for generators, to allow doing thinks like this:
$stream = new CallbackStream(function () {
    $command = new Command();
    $command->run();

    while ($command->isRunning()) {
        yield $command->getOutput(); //returns always a string
    }
    return false; //end of the command
});

In this case, the emitter should iterate with the stream, instead return all content at once. I mean:
while (!$stream->eof()) {
    echo $stream->read(1);
};

This behaviour is not implemented (currently $stream->read() throws a RuntimeException and the callback can be executed only once), but I can do it if you like.

In addition to my above comments, please run ./vendor/bin/phpcbf locally, and commit the results; you have CS errors, as reported here: https://travis-ci.org/zendframework/zend-diactoros/jobs/77199455#L175


SSO Authentication using Azure AD - Webchat Botbuilder V3

Hi all,
We are planning to integrate webchat created using Bot Builder V3 within an authenticated website.
For authentication, we are planning to use SSO of the website.
Please suggest the best practice to authenticate Chatbot users.
Thanks,
Kesavan Purushothaman

Take a look at this blog https://blog.botframework.com/2018/08/28/sign-in-experiences/ It should get you started.
Please post v3 questions in the v3 repo in the future.

Sorry, I do not have a working example to share.