[DesignProposal]: DataCollection API

Background and Motivation

For data scientists, towhee should accelerate their work, but stay close with their favorite toolkits;
For machine learning engineers, towhee should be scalable and as simple as a building block of the production system;

Features

Pythonic functional API;
Dispatch function to towhee operators; # TODO
Better error handling for large scale datasets; #649
Better performance by async execution; # TODO

Design
Design Goal:

Pythonic API: being natural to those python speakers;
Connective:  integrating well with python data science eco-system;

Core API
DataCollection provides a set of functional programming APIs that allows the user to assemble their data processing pipeline by chaining functions and operators. The method-chaining style API is proved to be very powerful in the area of data science and is widely accepted by the community.
Core API of DataCollection is designed to stay close with scala's and spark's collection API, with additional improvements for accelerating some work in python.

Creating

DataCollection is always created from Iterables
dc = DataCollection([ list | tuple | Iterator | Iterable ])

Combining small data collections will produce a larger one, this is useful if the data is collected from different sources:
small_dc_1 = DataCollection(range(10)) # [0...9]
small_dc_2 = DataCollection(range(10, 20)) #[10...19]
large_dc = small_dc_1 + small_dc_2 # [0...19]

Data collections can be ziped to a wider one:
dc_1 = DataCollection(range(10)) # [0...9]
dc_2 = DataCollection(range(10, 20)) #[10...19]

dc_1.zip(dc_2) # [(0, 10), (1,11)...(9,19)]

Zipping two data collection is useful when benchmarking a model on datasets, which needs predictions and labels to calculate model metrics.

Collection API

Collection API is a set of high-order functional APIs that apply function/operator to the collection and produce a new data collection.
map: apply function/operator to each elements and generate a new collection of the results
dc.map(lambda x: x+1) # apply a lambda function

multi-line function is also supported by decorator syntax:
@dc.map
def result(x):
    return x+1

Notice that function name is used as variable name for result data collection. In another word, result is a data collection, not a function.
filter: apply a filter to each element and generate a new collection that only contains filtered elements.
dc.filter(lambda x: x<3)

Advanced Collection API

dc = DataCollection(range(10))
dc.batch(5) # [[0,1,2,3,4], [5,6,7,8,9]]
dc.rolling(3) #[[0,1,2], [1,2,3], ..., [7,8,9]]
dc.batch(2).flaten() # [1, 2, 3, ..., 9], unbatch the data collection

Chaining functions

There are two ways for chaining functions/operators

chained collection API calls

dc
  .map(lambda x: x+1)
  .map(lambda x: x*2)

pipeline operator >>

(
dc
  >> lambda x: x+1
  >> lambda x: x*2
)

Use DataCollection with towhee
DataCollection support call dispatcher, which will redirect function on data collection to extended APIs. For examples:
@register_dispatcher
def add_1(x):
    return x+1

dc.add_1() # [add_1(x) for x in dc]

We can dispatch function calls to towhee operators, makes it much easier to assemble a towhee pipeline
dc = DataCollection(load_a_set_of_image())
dc.towhee.decode_image() # op_name = 'towhee/decode_image'
  .towhee.image_embedding(model_name='resnet101') # op_name = 'towhee/image_embedding'
  .towhee.vector_norm(norm='l2') # op_name = 'towhee/vector_norm'

Examples for building an image search engine with towhee

build image index

dc = DataCollection.glob('path/*.jpg')
dc.load_image()
  .towhee.image_embedding_pipeline(model='resnet50')
  .batch(128) # batched insert is usually more effective
  .to_milvus('https+milvus://host:port/db/db_name/table_name')

query

dc = DataCollection.from_files('query.jpg')
result = dc.load_image()
  .towhee.image_embedding_pipeline(model='resnet50')
  .query_milvus('https+milvus://host:port/db/db_name/table_name', top_k=20)
  .sortBy(lambda x: x.distance)
  .to_list()

Pros and Cons
No response
Anything else? (Additional Context)
No response

This API requires further review and discussion @GuoRentong @fzliu

@reiase Is there any way we can split the high level pythonic pipeline creation and the datacollection? I feel like a user should be able to create a pipeline without having to use the extra complexity that comes from datacollection.


Put support_type into Support table

Hi,
Its very hard to figure out if a row in the support table is actually just 'support' or really a 'tip'. My proposal would be to introduce support_type (I don't care which database type it will be, could be an int or enum or ...).
As long as you can easily distinguish if the support is a tip or a support (or whatever LBRY could have in the future as well).
Regards,
Melroy van den Berg

There's really no clean way to tell right now except for looking at the claim address, which historically may be different and could potentially change (guess we can still track at time of tx). The blockchain team is working on introducing metadata for supports, so we can then mark them appropriately among other cool things we can start to utilize the metadata for (i.e comments, ratings, subscriptions, signed supports)

Yeah, agreed with @tzarebczan here. There is no really accurate way to do this right now.


Access public variables of a class

Test Case https://codepen.io/daviuchoa/pen/jOWKNZX?editors=1010
Hi,
I'm using a simple class with a public instance variable called name and setting the class to the state. But a strange behavior occur when the method render()  is fired, the access of the name results in an error.
But if I comment the render() and after that set the state, it works.
Thanks in advance.

There seems to be some confusion on your end about how classes work.
The Hero class doesn't automatically get mapped to app.data.hero. When you run app.render(), props.hero has a value of null, because that's what you defined it as when setting up your component.
When you explicitly set app.data.hero to the output of new Hero('Iron Man'), you've now given it a value. The data update triggers a UI updates (because the data is reactive), and the UI updates with the new value.

You're right, I forgot about the reactivity concept in this case.
Thanks for the clarifications


upgrade-assistant analyze option is missing

Problem encountered on https://dotnet.microsoft.com/en-us/platform/upgrade-assistant/tutorial/install-upgrade-assistant
Operating System: windows
upgrade-assistant analyze option is not available.
upgrade-assistant analyze  outputs "Error: Unknown command 'analyze'."
Here is the output of  "upgrade-assistant" help
USAGE:
upgrade-assistant [OPTIONS]
EXAMPLES:
upgrade-assistant upgrade <PROJECT_PATH>
upgrade-assistant upgrade <PROJECT_PATH> --operation Inplace --targetFramework net6.0
upgrade-assistant upgrade <PROJECT_PATH> --operation SideBySide --targetFramework LTS --destination <NEW_PROJECT_NAME_OR_PATH>
OPTIONS:
-h, --help       Prints help information
-v, --version    Prints version information
COMMANDS:
upgrade    Upgrade project or its features. This command lets you pick upgrade options and guides through upgrade process

I have the same problem on Mac.

A temporary workaround for me is to uninstall and install the previous version.
dotnet tool install --global upgrade-assistant --version 0.4.421302

This is a duplicate of https://github.com/dotnet/upgrade-assistant/issues/1491

There are a lot of duplicates about the analyze command missing in the latest versions.
Starting with v0.5, the analyze command was removed.
Note: technically, it wasn't removed so much as the entire program was rewritten from scratch and the analyze command hasn't been implemented (yet?). In fact, this repo is obsolete and no longer contains the code for the upgrade-assistant tool. It was moved to another repo that gets pulled in by the Visual Studio build for better integration.

There are a lot of duplicates about the analyze command missing in the latest versions.
Starting with v0.5, the analyze command was removed.
Note: technically, it wasn't removed so much as the entire program was rewritten from scratch and the analyze command hasn't been implemented (yet?). In fact, this repo is obsolete and no longer contains the code for the upgrade-assistant tool. It was moved to another repo that gets pulled in by the Visual Studio build for better integration.

Ah, I looked closer now, and it turns out that Nuget also shows a different project "Project website" for this tool:
https://www.nuget.org/packages/upgrade-assistant/0.4.421302
--> Project website: https://github.com/dotnet/upgrade-assistant
https://www.nuget.org/packages/upgrade-assistant/0.5.261
--> Project website: https://go.microsoft.com/fwlink/?LinkID=2202641
I am not a fan of this kind of practice! I was not even sure that this was actually possible. But turns out it is.
This gives me the impression that Microsoft close sourced (!) this project from now on, and it's really becoming a proprietary Visual Studio-only tool.
That's why I do not like this.
However if this is only just a temporary situation, then let's leave it for what it is now.
But I would rather see that the new repository was already available to others before releasing new versions.
In that way others can keep contributing and help development, as it should be with an open source project.

Marking this as a duplicate of issue #1491


HMAC-SHA256 - Erroneous results

The procedure to calculate a HMAC-SHA256 (using flow: TEXT->HMAC/SHA-256->HASH) hash is invalid.
Test data
TEXT: 12345;78910
KEY: 12345
CORRECT HASH: 12476f110137a6c8243aa1ec839a4a09e7232ad682e407cc987ad3d1e2a91aef

INVALID!: bff8836fb60ed93f10677334fb04315776e381f82b72c8ab9b6a0373d2fdbd69
Check the screenshot.

Hi @mtsiakiris, the key setting is currently requested in hexadecimal format. Thus your key input 12345 is interpreted as hexadecimals instead of text. The result is correct. I understand that the UI may be misleading here, so I added the issue to the backlog. Thank's for reporting!


[v1.4.1] bump cryptography dep

Description of the change:
bump cryptography dep
Motivation for the change:
Checklist
If the pull request includes user-facing changes, extra documentation is required:

[ ] Add a new changelog fragment in changelog/fragments (see changelog/fragments/00-template.yaml)
[ ] Add or update relevant sections of the docs website in website/content/en/docs

/cherry-pick v1.3.x


[Event Hubs] remove createFromIotHubConnectionString method

Fixes #5203
Fixes #5219
This change removes the EventHubClient.createFromIotHubConnectionString method. Instead, an event-hubs compatible connection string can be used directly when instantiating an EventHubClient.

/azp run js - eventhubs-client - tests

/azp run js - eventhubs-client - tests


[Webpack 4] Percentage in ProgressPlugin is NaN

Bug report

What is the current behavior?
Prior to webpack 4, when running webpack-cli with --progress percentage was shown during the build. After upgrading to webpack 4, the percentage stopped appearing.
If the current behavior is a bug, please provide the steps to reproduce.
Run webpack-cli with --progress

What is the expected behavior?
Percentage should be shown through the build.

Other relevant information:
I narrowed the bug down to a change in the ProgressPlugin in this commit.
https://github.com/webpack/webpack/commit/f7c2f8e470d44686a2758c2debe59b86b3396c12
See ProgressPlugin.js
states[idx] was previously set to all the passed in arguments and now it is only set to the remaining arguments after pecentage and message. This causes the percentage to be NaN when the handler is called.
I left a comment in the commit.
webpack version:  4.8.1
Node.js version: 9.4
Operating System: Ubuntu 14.04

@evilebottnawi Here is a repo: https://github.com/Rohit-L/webpack-test
I have narrowed the problem down to when the webpack config returns an array of configurations. This causes the ProgressPlugin to go into the if (compiler.compilers) check, which is where the bug I described above lies.

@evilebottnawi Did you get a change to look at this. The fix should be very simple.

@Rohit-L WIP

@Rohit-L don't reproducible

All works fine

@evilebottnawi I got the list of process logs from test repo.
In the picture below, the left side is the current behavior, and the right side is the behavior after fixing the line that @Rohit-L mentions in the report.


Added authorization support for import/export.

https://jira.duraspace.org/browse/FCREPO-2163

@mikedurbin Do you want to squash, or shall I squash & merge?

The merge conflicts will be a little rough... I'll take care of it and force push shortly.


Declineoffer/getoffers returning no response/ not working

hi can you verify if the declineoffer/getoffers functionality are still working?
when i try to use it the callback only returns this value
"response: {}"
thanks

Yes, they are still working.

I am getting this issue, offers get sent i call the offerid with this method and get the same response. Its spastic though and lets some through while not others.


Added Date Table reference

In the Time-intelligence functions article, I added a reference to the Date Table documentation in the introductory paragraph.

@ankitk26 - Thanks for contributing. We can merge this, but please remove the link. The link you included is to an Analysis Services article. Because DAX is used with several different platforms, we don't want to send readers to a platform specific article.

@ankitk26 - Thanks for contributing. We can merge this, but please remove the link. The link you included is to an Analysis Services article. Because DAX is used with several different platforms, we don't want to send readers to a platform specific article.

@Minewiskan I have removed the link.


[windows] install ssh through msi

What does this PR do?
Install SSH on Windows Azure and AWS instances through the MSI
Which scenarios this will impact?
Windows
Motivation
Ensure compatibility across Windows versions, as Add-WindowsCapability installs different version depending on the windows version (Windows Server 2019 vs 2022)
Additional Notes
Suggested by @clarkb7
Tested with:

Windows Server 2019 ECS
Windows latest
Azure latest

@clarkb7 thank you for the suggestion, I added error handling to all the calls in the script, it would help debugging failing setups. Testing it again on the 3 AMIs I used for the first test

Tested against

aws custom AMI with ssh already installed
aws latest windows
aws Windows Server 2019
azure latest windows

/merge


Funny auto completion

I think this is related to #206 though the behavior slightly differs.

Thanks for reporting. I think this output looks different by coincidence rather than real change of behaviour. Let's fix the other one. Please follow that issue and if it doesn't fix your problem then we can re-open this one.


Pipe data from one extractor to another extractor

Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/185
Originally created by @bencodezen on 2018-11-27 20:00:28

cc @jschatz1

View 5 previous comments from the original issue on GitLab


[MANIFEST] Does not work with port-overlay feature.

Since vcpkg always throws an error if a port from an overlay is installed and the overlay is not fed into the vcpkg call.
Furthermore ports from an overlay can also not currently be installed since this is missing from the toolchain.
https://github.com/microsoft/vcpkg/blob/8133cef324060667d59949549eb4cdcf60b6699b/scripts/buildsystems/vcpkg.cmake#L278-L290
I would also suggest to move all code related to manifests into a separate cmake file for better separation from what the toolchain normally should do. (Manifests are just a modular addition to it.)

Is this a bug, or is it a feature request? Does this break anybody, or is it just that manifests don't work with overlay ports just yet?

The problem is if somebody tries to use manifests with a vcpkg which has an (unknown to master) port installed from a port-overlay vcpkg will throw and error on the execution of the command.
Has such you might not be immediately breaking someone but locally sharing vcpkg instances becomes a bit more difficult if you have projects which could use the manifest but have other projects which requires ports from an overlay.

This shouldn't happen, since the installed directories aren't shared -- if vcpkg/installed contains blah from an overlay, then proj/vcpkg_installed doesn't care about blah, since it's outside of the installed tree.

Closing since there is a way to define overlays via cmake now and registries will also solve this.


Tab control should select next tab when current tab is closed

Hey. I wanted to ask what behavior should be implemented if the current tab is the last one. Should the first tab in the list or the previous one be selected?

previous. use a browser's tab control as a guide for behaviour if unsure.

#3614

Resolved via #3614


Documentation not found for configuring a different backend like Graphite

Hi,
I wish to configure  Bosun with Graphite as a backend. Which is the configuration file I need to edit ?
https://bosun.org/configuration.html#backends
Thanks,
Shoubhik

Answers on Stack Overflow: http://stackoverflow.com/questions/31981042/integrate-graphite-metrics-with-bosun/31987567#31987567

Thank you very much Kyle.


Make Android SDK and buildtools versions the same for all exercise and examples

Depending on the exercise, there are different SDK versions defined. Namely between SDK versions 21 and 22. This requires additional SDKs to be downloaded. Since I didn't have Android Stedio setup beforehand, I had to do some trouble shooting when the SDK version was changed. This was a minor inconvenience, but I could see this being harder for new developers.
$ git grep -E "buildToolsVersion|compileSdkVersion"
3.03-Demo-TheDefaultAndroidBuildScripts/app/build.gradle:    compileSdkVersion 21
3.03-Demo-TheDefaultAndroidBuildScripts/app/build.gradle:    buildToolsVersion "21.1.2"
3.04-Demo-BuildTypes/app/build.gradle:    compileSdkVersion 22
3.04-Demo-BuildTypes/app/build.gradle:    buildToolsVersion "22.0.1"
3.05-Exercise-ConfigureBuildTypes/app/build.gradle:    compileSdkVersion 21
3.05-Exercise-ConfigureBuildTypes/app/build.gradle:    buildToolsVersion "21.1.2"
3.06-Demo-DeclaringFlavors/app/build.gradle:    compileSdkVersion 22
3.06-Demo-DeclaringFlavors/app/build.gradle:    buildToolsVersion "22.0.1"
3.07-Exercise-DeclareFlavors/app/build.gradle:    compileSdkVersion 21
3.07-Exercise-DeclareFlavors/app/build.gradle:    buildToolsVersion "21.1.2"
3.08-Exercise-AddAFlavor-SpecificActivity/app/build.gradle:    compileSdkVersion 22
3.08-Exercise-AddAFlavor-SpecificActivity/app/build.gradle:    buildToolsVersion "22.0.1"
3.08-Exercise-AddAFlavor-SpecificActivity/solution/app/build.gradle:    compileSdkVersion 22
3.08-Exercise-AddAFlavor-SpecificActivity/solution/app/build.gradle:    buildToolsVersion "22.0.1"
3.09-Demo-Flavor-SpecificDependencies/app/build.gradle:    compileSdkVersion 22
3.09-Demo-Flavor-SpecificDependencies/app/build.gradle:    buildToolsVersion "22.0.1"
4.01-Demo-CreatingAJavaLibrary/app/build.gradle:    compileSdkVersion 22
4.01-Demo-CreatingAJavaLibrary/app/build.gradle:    buildToolsVersion "22.0.1"
4.02-Exercise-CreateAJavaLibrary/solution/app/build.gradle:    compileSdkVersion 22
4.02-Exercise-CreateAJavaLibrary/solution/app/build.gradle:    buildToolsVersion "22.0.1"
4.03-Demo-CreatingAnAndroidLibrary/app/build.gradle:    compileSdkVersion 22
4.03-Demo-CreatingAnAndroidLibrary/app/build.gradle:    buildToolsVersion "22.0.1"
4.03-Demo-CreatingAnAndroidLibrary/imageactivity/build.gradle:    compileSdkVersion 22
4.03-Demo-CreatingAnAndroidLibrary/imageactivity/build.gradle:    buildToolsVersion "22.0.1"
4.04-Exercise-CreateAnAndroidLibrary/solution/app/build.gradle:    compileSdkVersion 22
4.04-Exercise-CreateAnAndroidLibrary/solution/app/build.gradle:    buildToolsVersion "22.0.1"
4.04-Exercise-CreateAnAndroidLibrary/solution/jokedisplay/build.gradle:    compileSdkVersion 22
4.04-Exercise-CreateAnAndroidLibrary/solution/jokedisplay/build.gradle:    buildToolsVersion "22.0.1"
4.05-Demo-AddingASigningConfiguration/app/build.gradle:    compileSdkVersion 22
4.05-Demo-AddingASigningConfiguration/app/build.gradle:    buildToolsVersion "22.0.1"
4.06-Exercise-ConfigureSigning/app/build.gradle:    compileSdkVersion 22
4.06-Exercise-ConfigureSigning/app/build.gradle:    buildToolsVersion "22.0.1"
4.07-Demo-MultidexSupport/app/build.gradle:    compileSdkVersion 22
4.07-Demo-MultidexSupport/app/build.gradle:    buildToolsVersion "22.0.1"
4.08-Demo-ConfiguringProguard/app/build.gradle:    compileSdkVersion 22
4.08-Demo-ConfiguringProguard/app/build.gradle:    buildToolsVersion "22.0.1"
4.09-Demo-AndroidUnitTesting/app/build.gradle:    compileSdkVersion 22
4.09-Demo-AndroidUnitTesting/app/build.gradle:    buildToolsVersion "22.0.1"
4.10-Exercise-SetUpUnitTesting/app/build.gradle:    compileSdkVersion 22
4.10-Exercise-SetUpUnitTesting/app/build.gradle:    buildToolsVersion "21.1.2"
4.10-Exercise-SetUpUnitTesting/solution/app/build.gradle:    compileSdkVersion 22
4.10-Exercise-SetUpUnitTesting/solution/app/build.gradle:    buildToolsVersion "21.1.2"
FinalProject/app/build.gradle:    compileSdkVersion 22
FinalProject/app/build.gradle:    buildToolsVersion "22.0.1"

Super good point, fixed in https://github.com/udacity/ud867/commit/79e46c1c92282a9b526ace7393fde6c937d8e5c7 and https://github.com/udacity/ud867/commit/b2a57e009641ff160cb7cbd6710b30cf3eeb2301
Thanks!


DS-2583: Quality Control Reporting via REST API (v4)

https://jira.duraspace.org/browse/DS-2583
Extend the REST API to support quality control reports.

GET /reports - Return a list of report tools built on the rest api
GET /reports/{nickname} - Return a redirect to a specific report
GET /filters - Return a list of use case filters available for quality control reporting
GET /filtered-collections - Return collections and item counts based on pre-defined filters
GET /filtered-collections/{collection_id} - Return items and item counts for a collection based on pre-defined filters
GET /filtered-items - Retrieve a set of items based on a metadata query and a set of filters

GET /reports
Configured in dspace/config/modules/rest.cfg
# return report tool html by name
#rest-report-url.collections = ../static/rest/index.html
#rest-report-url.item-query = ../static/rest/query.html

GET /filters
Configured in dspace/config/modules/rest.cfg.
These filters are referenced in the other reporting actions.
Each of these classes contains an enumerated list of useful ways to filter DSpace items.
# configurable filters for filter-collections and filtered-items
plugin.sequence.org.dspace.rest.filter.ItemFilterList = \
        org.dspace.rest.filter.ItemFilterDefs,\
        org.dspace.rest.filter.ItemFilterDefsMisc,\
        org.dspace.rest.filter.ItemFilterDefsPerm

Sample Reporting Application
The following JavaScript application illustrates how these new REST actions can provide quality control reporting to repository managers.
https://wiki.duraspace.org/display/DSPACE/REST+Based+Quality+Control+Reports

Did a preliminary review, will go a bit more in depth once these issues are resolved.

@KevinVdV, looking at the existing code in the rest package, most of the other classes do not log a stack trace.  Perhaps that is captured when a WebApplicationException is thrown.

@KevinVdV, following your suggestion, I changed the field list parameter in findByMetadataQuery() from List to List<List> and it simplified the query generation quite a bit.  Thank you for the suggestion.

@KevinVdV is this ready to merge? Are you comfortable merging it as-is?

I performed another review of the code (a bit more in depth, so after these fixes the code should be fine). I haven't tested any of it though, so I think once everything is fixed somebody else should take it for a test spin.
A couple of questions unrelated to the quality of the code:

Why use FilteredCollection as a return type why not Collection ?
Why use complex metadata queries to filter items & not use the discovery search framework ? This would heavily simplify the code to have a discovery endpoint & we wouldn't need the complex metadata fields ?
The last question is also directed towards the community, I wonder what other people think about this.

@KevinVdV , I appreciate the feedback on the code.  I will work up changes for your suggestions.
Since the notion of a filter is only applicable in the context of the new service calls, I did not want to complicate the Collection object with an unneeded filter object.
We use a precursor to these tools for metadata cleanup projects.  We use the output from these tools as input the to the bulk metadata edit process.  Our repository manager and collection owners want to craft highly controlled queries against the metadata fields.  Are the raw metadata fields clearly accessible in the SOLR index?  If 2 different metadata fields (dc.creator and dc.contributor.author) are indexed together in the discovery index can each field be queried individually?

@KevinVdV wrote:

Why use complex metadata queries to filter items & not use the discovery search framework ? This would heavily simplify the code to have a discovery endpoint & we wouldn't need the complex metadata fields ? The last question is also directed towards the community, I wonder what other people think about this.

You may need to check that this is even possible -- depending on the Solr field type (presumably stored vs not, tokenisation/normalisation, etc), we've found that it can be tricky to get Solr to do the right thing when it comes to retrieving the actual field value. I don't recall the details, sorry. Faceting was another problematic case I believe.

@KevinVdV , I have applied the changes that you suggested.  Thank you for the suggestions.

@tdonohue  suggested that I move the sample report app to DSpace-Labs.  I have updated the PR description to match that change.  Tim also requested that I improve the configuration documentation for this feature.  I will provide that update soon.

@tdonohue, I have added additional documentation to the config file and I have improved the report README.  I am open to any other suggestions that will be helpful to folks using this code.

@terrywbrady I had a brief look at the fixes & on a code level this is fine for me. Perhaps somebody else could aid me in testing the code ? (Since I don't know if I will have time for that in the near future).

@terrywbrady thanks! the comments in the config file look much improved.  I agree with @KevinVdV that this likely just needs someone else to test it out. Beyond that, looks good!

While performance testing this on a large database, I realized that the limit and offset parameters are not being used for the filtered-items query.
See my notes in https://jira.duraspace.org/browse/DS-2844 for a list of some of the database indexes that are likely needed to ensure that these requests are responsive.

I just ran this, and its hard to make much use out of the reports, since there are no reports included inside of this reporting framework. I've mapped /static in rest to a static directory, and pulled over the DSpace Rest Reports html, css, js reports into rest. This simplifies running this, and gives everyone running this reporting extension something to start with.
See: https://github.com/terrywbrady/DSpace/pull/3

@peterdietz , since the reports are now part of the pull request, I have migrated the documentation from DSpace-Labs to the DSpace Wiki: https://wiki.duraspace.org/display/DSPACE/REST+Based+Quality+Control+Reports
I also resolved some license header issues.
I am considering the following changes

remove the /rest/reports command now that the reports are deployed within the module
remove the README.md from the reports directory now that the documentation resides on the wiki
add documentation to describe how to disable this feature in web.xml

In attempting to test 1086, I pulled a copy of the PR and then attempted to rebase it on current master, I'm seeing merge conflicts:
both modified:      dspace-api/src/main/java/org/dspace/content/ItemServiceImpl.java
both modified:      dspace-api/src/main/java/org/dspace/content/dao/ItemDAO.java
both modified:      dspace-api/src/main/java/org/dspace/content/dao/impl/ItemDAOImpl.java
both modified:      dspace-api/src/main/java/org/dspace/content/service/ItemService.java

@hardyoyo, I presume that the merge requests are due to the length of time that this has been open.  I see those issues when I rebase locally.  What should I do to facilitate testing?

@tdonohue , @hardyoyo, I have rebased the branch and it should now merge for you.  Please let me know your test results.  I have done some limited testing since the rebase.  I need to reload the data in my test instance in order to run a thorough test.
If https://github.com/DSpace/DSpace/pull/1173 is adopted, there might be an opportunity to optimize the hierarchy retrieval with those new endpoints.

@hardyoyo pointed out that extraneous merges are present.  I will create a new branch and a new PR containing these changes.

Recreated this PR on a new branch.  See https://github.com/DSpace/DSpace/pull/1222


Model fix with no generic base

This fixes an issue when configured template base is not generic and the template uses @model

Sorry for the tardiness of my reply, been a busy few days. I'll look over this pull request later this week, with hope to action over the weekend.


Decode workers are created but never killed?

It seems like the decode workers are created each time a connection is initialized, but they are never killed. This (probably) is not good as it wastes computer resources during long debug sessions where you may reconnect many times.
You can observe this behavior by opening the devtools and then using a keybind to repeatedly connect to a robot/sim. You can see all the worker threads in the "Sources" tab:

Can a single decode worker be created once and reused between connections?

This issue is now fixed (c75f384) by creating each worker once on init and reusing them. See my comment on #7 as well. I'll make a new release soon to include this fix.


v1 auths do not seem to be working

i was working through an upgrade of influxdb 1.x and was not able to write data into the new 2.0 instance using the /write api and the v1 auth credentials.
to repro:

set up an influxdb 1.x instance from scratch using the default configuration options
load some data into that instance so you have something to upgrade.
shut down the 1.x instance and run the 2.0 influxd upgrade command in the latest RC4 binary
start influxdb 2.0
grab the token from the UI and use the influx v1 auth create command to create a new v1 auth
try to write data into 2.0 using the v1 auth you just made and the /write endpoint.

no matter what i try, i get the message:
{"code":"unauthorized","message":"Unauthorized"}
I was trying to do it this way:
curl -u russ:something --request POST http://localhost:8086/write\?db\=telegraf\&rp\=autogen --data-binary "measurement,host=host1 field1=2i,field2=2.0<PHONE_NUMBER>000000000"
{"code":"unauthorized","message":"Unauthorized"}

i verified the db/rp mapping is there from the dbrps endpoint:
curl --request GET http://localhost:8086/api/v2/dbrps\?orgID\=84369b2ddc1ebb8b \
--header "Authorization: Token =="
{
	"content": [
		{
			"id": "0693dfc08bbe8000",
			"database": "telegraf",
			"retention_policy": "autogen",
			"default": true,
			"organization_id": "84369b2ddc1ebb8b",
			"bucket_id": "18da3926266760db"
		}
	]
}

Can you provide the exact influx v1 auth create command you executed?

The problem is creating a token manually is a two-step process:

Create a token “foo”, which is equivalent to the username from 1.x:
influx v1 auth create --auth-token foo --write-bucket 7425a44ac4116e88 --read-bucket 7425a44ac4116e88

Assign a password to token “foo”, which creates the bcrypt hash:
influx v1 auth set-password --name foo

will be prompted for the password

The more I think about this, the UI for the CLI tool is just wrong. It was lifted from the v2 auth command, but should be clearer. We should change --auth-token to --username to be clearer from a 1.x perspective. Creating an auth token, by default, should prompt for a password and only create the token if it has both.

Verified auth is working when passwords are set, so closing this in favour of improving the influx v1 auth create command via #19933


Additional Maintainers Sought

Just want to be clear about that.
Anyway yeah, I could use help. There is a lot to do with this project and I want it to grow and be even better. So... I would love to bring on some more maintainers.
Here is what I propose:
The GDG Maintainer Program
Do you love fast as all heck data grids too? Great. We (really more of a royal we) are starting a 3 phase program to admit new maintainers:
Phase 1: Make at least 2 open-source code contributions
Past contributions count of course. Your contributions must either fix a bug in the core library or add a new feature. This shows some degree of familiarity with the project.
Phase 2: Documentation improvements
Several of the docs are out of date and need work. We need better release notes, up to date docs on new features and settings, ya know, grunt work. You can't spell "maintainer" without "free labor" I always say. This also shows willingness to work on the less than glorious parts of the project.
Phase 3: Co-PR review and issue management
You will review PR's submitted by me or other project contributors. We will work together to find a happy and healthy understanding of the goals and methods of this project. Prospective maintainers will also help to manage issues at this phase.
Once we have worked together long enough, I would like to promote at least one person to a core role. All new maintainers, at least for the time being will be at my sole discretion.

While I do intend to slowly hand over a greater degree of freedom to new maintainers, it will be a slow process. I don't want to let this project suffer mission creep and eventual death due to that.
If you want to help in a more limited capacity that is something of a hybrid between community contributor and maintainer let me know. I very much do need help here.

Also, the vaporwave website lives forever, sorry not sorry.
Let me know what you think! Consider this a PR to the project governance moving from a divine right dictatorship to an eventual divine right oligarchy.

(boring version to make sure non-native speakers don't get lost)
The Glide Data Grid (GDG) team is expanding and we are looking for new maintainers to join our project. We are introducing the GDG Maintainer Program, a structured approach to bring passionate individuals on board.
The GDG Maintainer Program
This program is a way to gradually integrate new maintainers into the project. It consists of three phases:
Phase 1: Code Contributions
To start, potential maintainers should make at least two open-source contributions to the project. These contributions can be either bug fixes or new features. This is to demonstrate familiarity with the core library.
Phase 2: Documentation Updates
The second phase involves updating and improving the project documentation. This includes revising outdated sections, enhancing release notes, and detailing new features and settings. This phase is critical as it requires attention to detail and a commitment to maintaining the project’s standards.
Phase 3: Collaborative Review and Issue Management
In this phase, candidates will collaborate on reviewing pull requests (PRs) and managing issues. This process is vital for understanding the project's goals and methodologies. It's a step towards a more significant role within the project.
Upon successful completion of these phases, we aim to promote at least one individual to a core maintainer role. The selection of new maintainers is at the discretion of the existing team.
Important Considerations

We are looking for individuals who are respectful and collaborative. Disruptive behavior is not conducive to the health of the project.
The transition to granting more autonomy to new maintainers will be gradual, ensuring the project's vision and integrity remain intact.
If you're interested in contributing but not ready for a full maintainer role, we welcome hybrid contributions between community contributor and maintainer.

The project's unique website design will remain as is for the foreseeable future.
We are open to feedback and ideas about this new governance approach. We aim to transition from a single-person-led project to a more collaborative team structure.

vaporwave 5eva

If you're interested in contributing but not ready for a full maintainer role, we welcome hybrid contributions between community contributor and maintainer.

I'm open to contribute free-time if needs be 😊

Can someone try to pick up this PR here: https://github.com/glideapps/glide-data-grid/pull/829


Profiler generates a missing title error message in the console

SQL Operations Studio Version: master

Fix is pending PR https://github.com/Microsoft/sqlopsstudio/pull/2176.


Cluster searches broken/incomplete

e.g.

Should be like VFB 1.5:

showing cluster image, exemplar etc

I guess I need to write a query -> JSON ?
Support info:
Tue Aug 20 2019 15:15:53 GMT 0800 (Taipei Standard Time)
Chrome 76.0.3809.87 [1073,681]
https://v2.virtualflybrain.org/org.geppetto.frontend/geppetto?id=FBbt_00003677&i=VFB_00017894,VFB_00030633,VFB_00004023
+ Passing instance: VFB_00004023_slices
+ Selection of Gad1-F-900045
+ Selection of 3D Skeleton
+ Selection of Gad1-F-900045
+ Selection of 3D Skeleton
+ Selection of Gad1-F-900045
+ Selection of 3D Skeleton
+ Selection of Gad1-F-900045
+ Selection of 3D Skeleton
+ Query requested: ImagesNeuronsClustered protocerebral bridge slice 9

Fixed by repairing Cluster label in pdb


Fixed log message

Tiny error

Yes %s also works but semantically %v seems more appropriate for errors. The problem however wasn't the usage of %s but that one of the referenced items was missing and logs were malformed

Update: I found another instance where the wrong formatting was used and the log was not displayed correctly


typescript-register throws error in gulp postinstall

Here's the full log from npm: http://cl.ly/1A2x163g1L12/npm-debug.log
Obviously the errors related to postinstall will be at the end.
angular2-seed/node_modules/typescript-register/index.js:132
    var result = checker.emitFiles();
                         ^

TypeError: checker.emitFiles is not a function

I'm assuming this is typescript-register's fault, but the build process is broken for this repo, so I thought I'd bring it up.
I think my internet connection may be flaking out on the build process, closing until I know for sure…
Yep, my own fault. Sorry for the false alarm :unamused:
Thanks for an awesome repo!

Hi, could you share your experience about solving this issue, please?
I have the same, but don't know what I need to do.
Thanks!

I think I just deleted and reinstalled. I can't say for sure tho…I use the Angular CLI now, which solved many of my issues from a few months ago


[2717] A couple of early years and opt-in undergrad snags

Context

https://trello.com/c/P2C18M2M/2717-snags-opt-in-and-early-years-ug

Changes proposed in this pull request

Fix award type label for opt-in ug
Fix missing translation for missing data

Guidance to review

 [Snags] Opt in and Early years UG


Do not force set ntp servers

There are cases where one does not want any ntp servers. Perhaps a node is using pools, or perhaps it's only using peers.
However, https://github.com/chef-cookbooks/ntp/pull/170 changed the behavior here. As
implemented in https://github.com/rhass/ntp/blob/master/recipes/default.rb#L61-L69
if node['ntp']['servers'].empty?
  node.default['ntp']['servers'] = [
    '0.pool.ntp.org',
    '1.pool.ntp.org',
    '2.pool.ntp.org',
    '3.pool.ntp.org',
  ]
  Chef::Log.debug 'No NTP servers specified, using default ntp.org server pools'
end

I'm not sure this was intentional or accidental.

Thanks for reporting. It looks like an unintended side effect of fixing this on mac os.

I might have misblamed #170, this may have always been true (which would represent a bug with my initial pools PR)
Regardless, I have fixed this in #178


Add GNOME Terminal

Checklist for submitting a pull request to Terminals Are Sexy:

[x] I have carefully read and comply with the Contributing Guidelines of this repo.
[x] I have searched the PRs of this repo and believe that this is not a duplicate.

Title: GNOME Terminal
Link: https://wiki.gnome.org/Apps/Terminal
Category: Terminal emulator
Description: A terminal emulator for GNOME.

I can endorse GNOME Terminal! 👍

Hey there @f2404. Please resolve conflicts with master. Thanks!

Done!

Thanks! Please also squash your commits into one.

Done.


Support sticker id to send

By LINE Notify API Documentation

Parameter name
Required/optional
Type
Description

message
Required
String
1000 characters max

imageThumbnail
Optional
HTTP/HTTPS URL
Maximum size of 240×240px JPEG

imageFullsize
Optional
HTTP/HTTPS URL
Maximum size of 2048×2048px JPEG

imageFile
Optional
File
Upload a image file to the LINE server.Supported image format is png and jpeg.If you specified imageThumbnail ,imageFullsize and imageFile, imageFile takes precedence.There is a limit that you can upload to within one hour.For more information, please see the section of the API Rate Limit.

stickerPackageId
Optional
Number
Package ID.Sticker List.

stickerId
Optional
Number
Sticker ID.

notificationDisabled
Optional
Boolean
true: The user doesn't receive a push notification when the message is sent.false: The user receives a push notification when the message is sent (unless they have disabled push notification in LINE and/or their device).If omitted, the value defaults to false.

Optional stickerId support needed

ID is invalid for official document
need to check it


using mapbox public key doesn't work

Following the instructions in the README, I set MAPBOX_ACCESS_KEY to the public key given. When I attempted to view the example mbtiles, I was met with a blank page. When I switched to using my private access key, the mbtiles were loaded. Is this issue specific to me or is the README outdated?
My steps using the public key:
npm install -g mbview
git clone git@github.com:mapbox/mbview
cd mbview/examples
export MAPBOX_ACCESS_TOKEN='pk.0000.1111'
mbview baja-highways.mbtiles

@jreiberkyle Did you replace pk.0000.1111 with a public access token from your account at https://www.mapbox.com/account/access-tokens?
pk.0000.1111 is just an example it's not a valid access token, you need to use a public access token from your own account.

@andrewharvey yep, and that worked!
I didn't realize from the documentation that pk.0000.1111 is just an example and that mbtiles requires my public access token to work. Part of the issue may be that it's not intuitive to me that a token would be required to view my own local mbtiles.

@jreiberkyle It's because mbview uses a Mapbox basemap as a background to your own local mbtiles. I'm think a PR to make that part optional, and hence work without a Mapbox access token would be welcomed.

@andrewharvey that is great and I'm sure that patch would be useful in that it would lower the barrier to using mbview! I took a quick look at the code and it looks like the style used to render vector features also requires a mapbox token. For now, I have created a PR (above, #80) that hopefully clarifies the need for a mapbox public token.

fixed in #80


E2E CI Creates a Second Job After Stopping First Job

This currently fails because this test is engineered to check for currently faulty behavior, and we should merge this once the queueing rewrite of the Switchboard is complete and fixes this behavior.

@charles37 The new commit you pushed is still based on the old revision, and the merge queue currently doesn't depend on the E2E test. However, I can change that, and then we can try merging this instead of rebasing.

@charles37 Can you rebase this test?

@charles37 Can you rebase this test?
ok doing so now.

@charles37 ping


Missing 'ID' attribute

The "ID" attribute is currently only added to the GVF by addfunctionalinfo2gvf.py, because it's based on the comb_mutation attribute from the functional annotations file.  It shouldn't affect the visualization to have it empty (I think--I need to check with Ivan) but I will need to update it to fill in an ID for all GVF outputs because it's required as part of the standard GVF format.  This means that the IDs may not be consistent through different versions of the same GVF.

-"ID" attribute is unused in visualization
-updated ID in addfunctions2gvf.py in 134beeb


operator: Add cilium node garbage collector

Description
In the normal scenario, CiliumNode is created by agent with owner
references attached all time time in below PR[0]. However, there could
be the case that CiliumNode is created by IPAM module[1], which
didn't have any ownerReferences at all. For this case, if the
corresponding node got terminated and never came back with same
name, the CiliumNode resource is still dangling, and needs to be
garbage collected.
This commit is to add garbage collector for CiliumNode, with below
logic:

Gargage collector will run every predefined interval (e.g. specify
by flag --nodes-gc-interval)
Each run will check if CiliumNode is having a counterpart k8s node
resource. Also, remove this node from GC candidate if required.
If yes, CiliumNode is considered as valid, happy day.
If no, check if ownerReferences are set.

If no, mark the node as GC candidate. If in the next run, this
node is still in GC candidate, remove it.

Testing
Testing was done locally with kind cluster

Create one dummy ciliumnode and check gc
$ ksyslo deployment/cilium-operator --timestamps | grep kind-worker-dummy
2022-04-26T10:59:22.106774489Z level=debug msg="UpdatedStatus Node" error="<nil>" node-name=kind-worker-dummy subsys=pod-cidr
2022-04-26T10:59:23.087007602Z level=debug msg="UpdatedStatus Node" error="<nil>" node-name=kind-worker-dummy subsys=pod-cidr
2022-04-26T10:59:51.977272944Z level=info msg="Add CiliumNode to garbage collector candidates" nodeName=kind-worker-dummy subsys=watchers
2022-04-26T11:00:21.978387751Z level=info msg="Perform GC for invalid CiliumNode" nodeName=kind-worker-dummy subsys=watchers

$ kg ciliumnodes
NAME                 AGE
kind-control-plane   156m
kind-worker          156m

Thanks for your input, indeed there are a few cases I didn't consider.

Seeing that we are selectively adding RBAC permissions and passing the nodes-gc-interval flag through the ConfigMap, I'm assuming there is a use-case for disabling this feature. In what context users would want to disable it?

Normally, we don't want to disable this GC, it's just an option for users in case of any issue later on.

Settting to null shows nothing, meaning we don't pass the flag and we don't have the RBAC permission. The issue here is that we'll be using the non-zero CLI default (5m) and will fail because we don't have the permission.
Settting to empty will actually pass it as-is to the flag, which will fail duration parsing.
Settting to 0s will correctly disable the feature in the ConfigMap, but won't add the RBAC permission.

Agreed on the above points, I have made below changes to rectify

update default value in CLI to 0s i.e disabled by default, so that we don't couple with RBAC permission
add default value in config map
add check in RBAC to make sure that we only add permission if .Values.operator.nodeGCInterval is having some value, and it's not 0s. I don't think we need to check for correct data type (e.g. duration) in helm.

$ helm template cilium ./install/kubernetes/cilium --namespace kube-system --set operator.nodeGCInterval=null | grep -E "nodes-gc-interval|# To perform CiliumNode garbage collector"

$ helm template cilium ./install/kubernetes/cilium --namespace kube-system --set operator.nodeGCInterval= | grep -E "nodes-gc-interval|# To perform CiliumNode garbage collector"
  nodes-gc-interval: "0s"

$ helm template cilium ./install/kubernetes/cilium --namespace kube-system --set operator.nodeGCInterval=0s | grep -E "nodes-gc-interval|# To perform CiliumNode garbage collector"
  nodes-gc-interval: "0s"

$ helm template cilium ./install/kubernetes/cilium --namespace kube-system --set operator.nodeGCInterval=1s | grep -E "nodes-gc-interval|# To perform CiliumNode garbage collector"
  nodes-gc-interval: "1s"
    # To perform CiliumNode garbage collector

/test

/test-1.23-net-next

Thanks Alex.
I noticed that this suggestion should be applied for other duration field in helm values as well. For consistency, I think we can just keep as what we have in this PR.
Raised one https://github.com/cilium/cilium/issues/19684 for future improvement.

All tests passed except NAT-46x64 and ci-l4lb jobs are still under waiting, there two jobs are going to merge to one, and they are not related to this change.
Mark PR ready to merge.

waiting on https://github.com/cilium/cilium/pull/19686 to be merged

Force push to resolve the conflict with master, only one file is changed, the diff can be found as per below:
diff --git a/install/kubernetes/cilium/templates/cilium-operator/clusterrole.yaml b/install/kubernetes/cilium/templates/cilium-operator/clusterrole.yaml
index 531566cc4b..8af8b9824b 100644
--- a/install/kubernetes/cilium/templates/cilium-operator/clusterrole.yaml
+++ b/install/kubernetes/cilium/templates/cilium-operator/clusterrole.yaml
@@ -21,21 +21,12 @@ rules:
 {{- end }}
 {{- end }}
 {{- end }}
-{{- if or .Values.operator.removeNodeTaints .Values.operator.setNodeNetworkStatus .Values.operator.endpointGCInterval }}
-- apiGroups:
-  - ""
-  resources:
-  - nodes
-  verbs:
-  - list
-  - watch
-{{- end }}
 {{- if or .Values.operator.removeNodeTaints .Values.operator.setNodeNetworkStatus }}
 - apiGroups:
   - ""
   resources:
-{{- if .Values.operator.removeNodeTaints }}
-  # To remove node taints
+{{- if or .Values.operator.removeNodeTaints .Values.operator.nodeGCInterval }}
+  # To remove node taints or perform underlying CiliumNode garbage collection
   - nodes
 {{- end }}
 {{- if .Values.operator.setNodeNetworkStatus }}
@@ -43,6 +34,8 @@ rules:
   - nodes/status
 {{- end }}
   verbs:
+  - list
+  - watch
   - patch
 {{- end }}
 - apiGroups:

/test

/test-gke

/test-1.16-4.9

/test-1.22-4.19

/test-1.23-net-next

/test-1.21-5.4

All the tests passed now, marking this ready for merge.


v-if not working when used with v-for in child component

https://jsfiddle.net/vehj65vv/4/

v-if="member.channelName !== channelName"

channelName is undefined in your case. You forgot to add it data object

Please make sure to read the Issue Reporting Guidelines before opening new issues. The issue list only accepts bug reports and feature requests. Questions should be posted to the Gitter chat room,  forum or StackOverflow. Thanks.
It does work: https://jsfiddle.net/posva/gw02w42o/
But as a side note I recommend you using a computed to filter the array first instead of using an if 😉


pip install 2nd time install failed

pip install . => ok (setuptools.command.egg_info is handled)
pip install -e . => ok (setuptools.command.develop is handled)
pip install rpm-py-installer => 1st time: ok 2nd time: ng (the installer logic is not run.)
Maybe there is the another handler.
Maybe 1st: setuptools.command.install is handled.
So, how about 2nd? (cached status?)

Ref: https://github.com/junaruga/rpm-py-installer/blob/master/setup.py

Maybe this is a reason.
https://stackoverflow.com/questions/40433168/running-custom-code-with-pip-install-fails

you need to publish you package as a source distribution (sdist) and not a wheel (bdist_wheel), because setup.py is not executed during install as a wheel

I have already uploaded as sdist.
The reason was cache.
The case that rpm-python is not installed was cache used case.
$ pip install -vvv rpm-py-installer
...
  Using cached wheel link: file:///home/jaruga/.cache/pip/wheels/9c/97/e9/6b5c91b77f2e2c9fd73528c4840ed7d4625d5c1ee423f05cfa/rpm_py_installer-0.1.0-cp36-none-any.whl
Installing collected packages: rpm-py-installer

Below operation works.
$ which rpm
/usr/bin/rpm

$ rpm --version
RPM version <IP_ADDRESS>

$ rm -rf ~/.cache/pip

(venv) $ pip list
Package    Version
---------- -------
pip        9.0.1--
setuptools 28.8.0-

(venv) $ pip install rpm-py-installer

(venv) $ pip list
Package          Version
---------------- --------
pip              9.0.1
rpm-py-installer 0.1.0
rpm-python       <IP_ADDRESS>
setuptools       28.8.0

How to run custom install with hundling cached wheel trigger?

Way 1: Set --no-cache-dir option in setup.py forcely.
Way 2: Control cache manually. Ref https://github.com/authomatic/gae_installer/blob/master/setup.py#L45

wheel package is installed, the cache is used. This issue is happened.
But if wheel is not installed on a python environment, this issue is not happened. No problem.

Finally fixed by https://github.com/junaruga/rpm-py-installer/pull/46


Trust store password is incorrect following a bosh restart

What version of the credhub server you are using?
1.9.3
What version of the credhub cli you are using?
N/A
If you were attempting to accomplish a task, what was it you were attempting to do?
Trying to do a bosh restart on a fresh deployment of concourse.
What did you expect to happen?
We expected the job to restart successfully
What was the actual behavior?
The job fails for credhub and the following output is produced.
Using environment '<environment>' as client 'admin'

Task 47

Task 47 | 09:51:25 | Preparing deployment: Preparing deployment (00:00:01)
Task 47 | 09:51:29 | Preparing package compilation: Finding packages to compile (00:00:01)
Task 47 | 09:51:30 | Updating instance web: web/3e75a779-5735-42b9-b025-5983eff2e72b (0) (canary) (00:12:03)
                   L Error: Action Failed get_task: Task 3f8ac090-4793-43f0-67a5-75d96aa80d80 result: 1 of 3 post-start scripts failed. Failed Jobs: credhub. Successful Jobs: uaa, grafana.
Task 47 | 10:03:33 | Error: Action Failed get_task: Task 3f8ac090-4793-43f0-67a5-75d96aa80d80 result: 1 of 3 post-start scripts failed. Failed Jobs: credhub. Successful Jobs: uaa, grafana.

Task 47 Started  Wed Jun 13 09:51:25 UTC 2018
Task 47 Finished Wed Jun 13 10:03:33 UTC 2018
Task 47 Duration 00:12:08
Task 47 error

Capturing task '47' output:
  Expected task '47' to succeed but state is 'error'

Exit code 1

After investigating further we found the following in the credhub.log
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780) ~[?:1.8.0_172]
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) ~[?:1.8.0_172]
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_172]
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) ~[?:1.8.0_172]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_172]
        at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226) ~[?:1.8.0_172]
        at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:771) ~[?:1.8.0_172]
        at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(SSLContextImpl.java:748) ~[?:1.8.0_172]
        at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:913) ~[?:1.8.0_172]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_172]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_172]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_172]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_172]
        at java.security.Provider$Service.newInstance(Provider.java:1595) ~[?:1.8.0_172]
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) ~[?:1.8.0_172]
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) ~[?:1.8.0_172]
        at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) ~[?:1.8.0_172]
        at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) ~[?:1.8.0_172]
        at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122) ~[?:1.8.0_172]
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:44) ~[postgresql-42.2.1.jar!/:42.2.1]
        ... 48 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778) ~[?:1.8.0_172]
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56) ~[?:1.8.0_172]
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224) ~[?:1.8.0_172]
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70) ~[?:1.8.0_172]
        at java.security.KeyStore.load(KeyStore.java:1445) ~[?:1.8.0_172]
        at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226) ~[?:1.8.0_172]
        at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getTrustManagers(SSLContextImpl.java:771) ~[?:1.8.0_172]
        at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.<clinit>(SSLContextImpl.java:748) ~[?:1.8.0_172]
        at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:913) ~[?:1.8.0_172]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_172]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_172]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_172]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_172]
        at java.security.Provider$Service.newInstance(Provider.java:1595) ~[?:1.8.0_172]
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) ~[?:1.8.0_172]
        at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) ~[?:1.8.0_172]
        at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) ~[?:1.8.0_172]
        at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) ~[?:1.8.0_172]
        at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122) ~[?:1.8.0_172]
        at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:44) ~[postgresql-42.2.1.jar!/:42.2.1]
        ... 48 more
2018-06-13T09:53:31.295Z [main] .... ERROR --- ConnectionPool: Unable to create initial connections of pool.
org.postgresql.util.PSQLException: SSL error: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

This suggested that the password was wrong for the trust store. We found that the password displayed in the config/application.yml did not match the one used to launch the server in the ctl script.
web/3e75a779-5735-42b9-b025-5983eff2e72b:/var/vcap/jobs/credhub# cat config/application.yml | grep trust
  trust_store: /var/vcap/jobs/credhub/config/trust_store.jks
  trust_store_password: Mt5WbtjoSlgku0wioEpNrk5oxOZ47iZg

the trust_store_password does not match:
web/3e75a779-5735-42b9-b025-5983eff2e72b:/var/vcap/jobs/credhub# cat bin/ctl | grep trust
-Djdk.tls.namedGroups="secp384r1" \-Djavax.net.ssl.trustStore=/var/vcap/jobs/credhub/config/trust_store.jks \
  -Djavax.net.ssl.trustStorePassword=0olVqtPifTqfqmGPaHYRxFhBEE315deA \-ea -jar *.jar

After debugging further, we found that the init_key_stores.sh which is called by the pre-start script does not successfully substitute the generated trust store password for the TRUST_STORE_PASSWORD_PLACEHOLDER. If the pre start script has already run then no newly generated passwords will be substituted into these files as the placeholder will not be there.
We managed to get this working by manually replacing the old passwords with the placeholder before running bosh restart again.
Please confirm where necessary:

[x] I have included a log output
[x] My log includes an error message
[x] I have included steps for reproduction

If you are a PCF customer with an Operation Manager (PCF Ops Manager) please direct your questions to support (https://support.pivotal.io/)

Just happened to hit this issue as well with CredHub 1.9.5. Symptoms are slightly different, even worse than described above:

deploying bosh didn't fail, all jobs started successfully
once we tried to deploy anything that tried to use CredHub, this fails very silently with invalid token

The pre-start script is not idempotent, although it contains a check that tries to achieve this:

configurator always writes a new application.yml with the placeholder KEY_STORE_PASSWORD_PLACEHOLDER: https://github.com/pivotal-cf/credhub-release/blob/master/jobs/credhub/templates/pre-start.erb#L8
Check for 'is truststore already set up?' always returns 'false': https://github.com/pivotal-cf/credhub-release/blob/master/jobs/credhub/templates/pre-start.erb#L38-L39
init_key_stores.sh deletes existing keystore and creates one with a new password
if templates had not been re-rendered by the director (and therefore contain the placeholder KEY_STORE_PASSWORD_PLACEHOLDER instead of the previous password), ctl and credhub now contain the old password, while the keystore has a new password

suggested solution: make pre-start idempotent, as suggested by bosh documentation: https://bosh.io/docs/pre-start/
Unfortunately, this seems so happen quite often now, most notably on AWS. I'm not sure yet why this is the case right now, but this makes it pretty awful for us.
This bug has been open for about two weeks without any acknowledgement or reaction – is this something you would be able to prioritze?

For reference: here is what we ended up doing for our fork for now. Just in case someone has the same problem and needs to fix this right now.
From 85bcfa210bd942e9c6c8c81d58cd4d0a2b9c37f8 Mon Sep 17 00:00:00 2001
Date: Mon, 30 Jul 2018 16:45:26 +0200
Subject: [PATCH] make pre-start idempotent
---
 jobs/credhub/templates/pre-start.erb | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/jobs/credhub/templates/pre-start.erb b/jobs/credhub/templates/pre-start.erb
index aaf41ca..4dd970d 100644
--- a/jobs/credhub/templates/pre-start.erb
+++ b/jobs/credhub/templates/pre-start.erb
@@ -3,9 +3,6 @@
 <% using_hsm = p('credhub.encryption.providers').any? { |provider| provider['type'] == "hsm" } %>
 <% port = p('credhub.port') %>

-
-cat /var/vcap/jobs/credhub/config/config.json | /var/vcap/packages/configurator/bin/configurator/configurator > /var/vcap/jobs/credhub/config/application.yml
-
 <%
   begin
     p('credhub.tls.certificate')
@@ -36,10 +33,11 @@ export JAVA_HOME=/var/vcap/packages/openjdk_1.8.0/jre
 chown -R vcap /var/vcap/jobs/credhub/config
 chmod 600 /var/vcap/jobs/credhub/config/*

-if [ `grep KEY_STORE_PASSWORD_PLACEHOLDER /var/vcap/jobs/credhub/config/application.yml | wc -l` -eq 0 ]; then
+if [ `grep TRUST_STORE_PASSWORD_PLACEHOLDER /var/vcap/jobs/credhub/bin/ctl| wc -l` -eq 0 ]; then
   echo "Key store appears to be initialized. Skipping key store and cert initialization."
 else
   echo "Key store appears to be uninitialized. Setting up key store and installing certs."
+  cat /var/vcap/jobs/credhub/config/config.json | /var/vcap/packages/configurator/bin/configurator/configurator > /var/vcap/jobs/credhub/config/application.yml
   /var/vcap/jobs/credhub/bin/init_key_stores.sh
 fi

Nope, no HSM being used. Would that change anything about the script being idempotent?

Shouldn't; just thought it might be a factor in the AWS error-tendencies.

Underlying issue why the credhub pre-start script got executed multiple times on AWS:
https://github.com/cloudfoundry/bosh/issues/2017

@iferunsewe I think we fixed this issue here but never cut a 1.9.x release including it. It's in 2.0, but we'll also back-port in case you aren't ready to upgrade to that version yet.
@voelzmo We weren't able to reproduce your issue using 1.9.5, do you think the above commit would fix it? We were a little confused by the last step of your reproduction:

if templates had not been re-rendered by the director (and therefore contain the placeholder KEY_STORE_PASSWORD_PLACEHOLDER instead of the previous password), ctl and credhub now contain the old password, while the keystore has a new password

What do you mean by the templates not being re-rendered? The init_key_stores.sh script is only ever called by the pre-start script, which ensures the placeholders are put into the application.yml. The check to see if the key store is initialized is unnecessary, but the script itself seems idempotent.

pre-start is not idempotent, you should be able to easily test this with

bosh deploy/create-env credhub
ssh into the machine, see that passwords in application.yml and the ctl are the same
monit stop credhub
execute /var/vcap/jobs/credhub/bin/pre-start
see that passwords in application.yml and the ctl are different
check that the keystore password that works is the one from application.yml, as it gets re-generated

You can repeat this as often as you want, the keystore gets a new password on every execution of pre-start.
Above, I tried to explain why this is the case, and how the scenario of pre-start being executed twice happens. Sorry if this didn't come across quite right.

@voelzmo The fix for this issue is in 1.9.6 and 2.0.1. The trust store password still gets changed in the pre-start, but we properly update it in all of the files.

@mdelillo Thanks for letting me know! I'm a bit confused about the fix that you decided for, though. Could you help me understand why you decided to not make the script idempotent but change the way the replacement happens, such that there's no mismatch between the files containing the password?

@voelzmo We change the password every time because it's the simplest thing for us to do and was the smallest change that fixed the issue. I suppose we could have generated the passwords once in another file and checked for the presence of that file to determine whether to regenerate them or not, but we didn't see much advantage to doing that over our solution. We would still want to recreate the trust stores every time the script is run, or at the very least add the certificates to the existing ones (in case the certs have changed).


Aemet sensor

Description:
This component adds sensor support for AEMET's wheather stations in Spain by using their API service (https://opendata.aemet.es)
Pull request in home-assistant.io with documentation (if applicable): home-assistant/home-assistant.io#7920
Example entry for configuration.yaml (if applicable):
sensor:
  - platform: aemet
    name: Retiro station
    api_key: YOUR_API_KEY
    station_id: 3195
    monitored_conditions:
      - temperature
      - humidity
      - pressure
      - precipitation
      - snow
      - visibility

Checklist:

[x] The code change is tested and works locally.
[x] Local tests pass with tox. Your PR cannot be merged unless tests pass
[x] There is no commented out code in this PR.

If user exposed functionality or configuration variables are added/changed:

[x] Documentation added/updated in home-assistant.io

If the code communicates with devices, web services, or third-party tools:

[ ] New dependencies have been added to the REQUIREMENTS variable (example).
[ ] New dependencies are only imported inside functions that use them (example).
[ ] New or updated dependencies have been added to requirements_all.txt by running script/gen_requirements_all.py.
[x] New files were added to .coveragerc.

If the code does not interact with devices:

[ ] Tests have been added to verify that the new code works.

Hi @kalanda,
It seems you haven't yet signed a CLA. Please do so here.
Once you do that we will be able to review and accept this pull request.
Thanks!

There is a discussion that new meteorological integrations should only be weather platforms and no longer sensor platforms.
https://github.com/home-assistant/architecture/issues/107

@fabaff weather over sensor is a requirement for this PR or a suggestion? This component doesn't get information about weather forecast, it only gets information about conditions measurements across the country stations

@kalanda weather is a requirement


Consul 0.4 bug causes docker swarm manager to crash

I installed swarm using the instructions from
https://www.upcloud.com/support/how-to-configure-docker-swarm/
in short:
sudo service docker stop
sudo rm nohup.out
sudo nohup docker daemon -H tcp://<IP_ADDRESS>:2375 -H unix:///var/run/docker.sock &
sleep 5
docker run -d -p 8500:8500 --name=consul progrium/consul -server -bootstrap
docker run -d -p ${MASTER_IP}:4000:4000 swarm manage -H :4000 --replication --advertise ${MASTER_IP}:4000 consul://${MASTER_IP}:8500

Consul Crashed
Log of container:
2 errors:

panic: Timeout starting MDB transaction, potential deadlock
WARNING: It is highly recommended to set GOMAXPROCS higher than 1

Helpful links

panic: https://github.com/hashicorp/consul/issues/1239
GOMAXPROCS warning: https://groups.google.com/forum/#!msg/consul-tool/qewFEqgAoF8/b9hxhmy1v6gJ

Cause of error:

panic: bug in consul in version 0.4; this bug is removed in version 0.6

Resolution:

run docker consul image of version 0.6
problem: official docker image of consul does not support versions higher than 0.4
I used another docker image: https://hub.docker.com/r/gliderlabs/consul/
Docker run command to start consul: The agent command must be explicitly added to docker, -data-dir option is mandatory and -client=<IP_ADDRESS> fixes the problem that consul docker container accepts connections from localhost or OpenStack IP address:

docker run -d -p 8500:8500 --name=consul gliderlabs/consul:0.6  agent -server -bootstrap -data-dir=/data/consul -client=<IP_ADDRESS>

According to the site Swarm doesn't support consul versions < 0.5.1

Ok it seems I used an outdated tutorial. I will close this issue then.

Actually, I looked for the instructions on the site:
https://docs.docker.com/swarm/install-manual/
This also uses the progrium/consul docker image. However this docker image does not support consul versions higher than 4.0

I'm running that image right now and it's at version 0.5.2 for me.

Then you build it from a modified Dockerfile.
The dockerfile in the dockerhub repository still uses 0.4
https://hub.docker.com/r/progrium/consul/~/dockerfile/:

If you run docker run -d --name consul -p 8400:8400 -p 8500:8500 -p 8600:53/udp -h node1 progrium/consul -server -bootstrap you will be running 0.5.2.  To verify that, run that command and then run docker exec -it consul consul members and it will show you that you're using build 0.5.2.

Ok I trust you :)
Still that container image crashed frequently on my system
Can you please check the logs of your  consul container for panic: Timeout starting MDB transaction, potential deadlock?


fix(behavior_velocity_planner): avoid duplicates in intersection module

Signed-off-by: Maxime CLEMENT<EMAIL_ADDRESS>Description

The intersection module of the behavior_velocity_planner outputs very large debug message when crossing multiple intersections. Moreover, the CPU load also greatly increases in that case.
This PR solves this issue by simplifying the detection_area_lanelets calculated by function getObjectiveLanelets().
This variable is used to (1) calculate the lanelet IDs to use for collision detection and (2) calculate the debug polygons to highlight in rviz. Originally, the detection_area_lanelets was a vector of sequences of lanelets where the sequences could overlap, sharing many common lanelets. This would result in many duplicated IDs (1) and many overlapping polygons (2).
In this PR, the detection_area_lanelets is changed to be a vector of lanelets without any duplicate, which simplifies both (1) and (2) without changing the behavior of the module.
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[X] I've confirmed the contribution guidelines.
[X] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

@maxime-clem Thank you! I asked @taikitanaka3 to review this PR.

I tested with


Error  "....is not a valid UUID." in checkoutComplete and orderCapture (Saleor 3.4.5)

What I'm trying to achieve
I am trying to make purchases in Saleor using the Dummy plugin (Saleor 3.4.5).
However, I get this error but the order is indeed created!
{
  "data": {
    "checkoutComplete": {
      "confirmationNeeded": false,
      "confirmationData": null,
      "errors": [
        {
          "field": null,
          "message": "“T3JkZXI6MjAzYzdkOTItY2Y4YS00Nzk3LWEyMTEtOGE5MDQ0YjdiNGFi” is not a valid UUID.",
          "code": "INVALID"
        }
      ]
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 0,
      "maximumAvailable": 50000
    }
  }
}

Where T3JkZXI6MjAzYzdkOTItY2Y4YS00Nzk3LWEyMTEtOGE5MDQ0YjdiNGFi is the id of the order
Steps to reproduce the problem

Perform a checkoutCreate
Perform checkoutPaymentCreate with gateway: "mirumee.payments.dummy"
Perform completeCheckout

What I expected to happen
I expect the checkoutComplete mutation to finish without errors and creating the order.
Screenshots and logs
Response of checkoutComplete mutation:
{
  "data": {
    "checkoutComplete": {
      "confirmationNeeded": false,
      "confirmationData": null,
      "errors": [
        {
          "field": null,
          "message": "“T3JkZXI6MjAzYzdkOTItY2Y4YS00Nzk3LWEyMTEtOGE5MDQ0YjdiNGFi” is not a valid UUID.",
          "code": "INVALID"
        }
      ]
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 0,
      "maximumAvailable": 50000
    }
  }
}

Logs: I don't see the error in logs, however, trying to debug with pdb.set_trace(), I managed to see this error:
django.core.exceptions.ValidationError: ['“T3JkZXI6MjAzYzdkOTItY2Y4YS00Nzk3LWEyMTEtOGE5MDQ0YjdiNGFi” is not a valid UUID.']

System information

Saleor version:

[ ] dev (current main)
[x] 3.4.5
[ ] 3.0
[ ] 2.11
[ ] 2.10

Operating system:

[ ] Windows
[x] Linux (Docker)
[ ] MacOS
[ ] Other

It was a problem in one of our custom plugins


adding new column message in kctrl pkg installed list command

What this PR does / why we need it:
Adding a new column 'Message' in the output of command kctrl package installed list, which will show the message from last condition from the status of each package install when --wide flag is provided.
The sample output from testing.
 % /tmp/kctrl package installed list
Target cluster 'https://<IP_ADDRESS>:50874' (nodes: local-test-control-plane)

Installed packages in namespace 'default'

Name          Package Name         Package Version  Status
pkg-demo      simple-app.corp.com  1.0.0            Reconcile succeeded
pkg-demo-err  simple-app.corp.com  1.0.0            Reconcile failed

Succeeded
 % /tmp/kctrl package installed list --wide
Target cluster 'https://<IP_ADDRESS>:50874' (nodes: local-test-control-plane)

Installed packages in namespace 'default'

Name          Package Name         Package Version  Status               Message
pkg-demo      simple-app.corp.com  1.0.0            Reconcile succeeded  -
pkg-demo-err  simple-app.corp.com  1.0.0            Reconcile failed     Error (see .status.usefulErrorMessage for details)

Succeeded

Which issue(s) this PR fixes:

Fixes #
Does this PR introduce a user-facing change?

Adds a new column 'Message' in the output of command `kctrl package installed list`, which will show the message from last condition from the status of each package install when --wide flag is provided

Additional Notes for your reviewer:
Review Checklist:

[ ] Follows the developer guidelines
[ ] Relevant tests are added or updated
[ ] Relevant docs in this repo added or updated
[ ] Relevant carvel.dev docs added or updated in a separate PR and there's
a link to that PR
[ ] Code is at least as readable and maintainable as it was before this
change

Additional documentation e.g., Proposal, usage docs, etc.:

Do you think we should keep the column hidden by default and only display it when any one of the pkgi is failing? (I am okay if we want to think about that later on based on feedback from users)

so moved the new column under --wide flag.


Wiki formatting issue

There's an issue with the Wiki FAQ. Add a space between ### and OSX unless it's intentionally trying to show that Apple deserves no respect.

Fixed, ty


Start improving the documentation

This PR fixes #3 by adding the minimum documentation for interested users.
More in-depth documentation (e.g., API documentation) is still missing, but will be added slowly in the future.

I'm to slow :cry:

Maybe open a new PR with the review content? :)

Will do, It's just a couple of typos, though


plugin-torture: 2016-07-25 -> 5

Motivation for this change
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[x] Fits CONTRIBUTING.md.

This appears to be an older version than the commit at 2016-07-25. Is that intentional?

@srhb Good catch.  I saw a version hash instead of a nr, and just assumed the pkg was still in the version I packaged ages ago.


Unable to created Project

Error while creating release definition: Tasks with versions 'ARM Outputs:6.*' are not valid for deploy job 'Agent job' in stage Stage 1.
AB#3872

Kindly install the ARM Outputs task manually from Marketplace.
Please refer to #626


Generate config file (not output) with mas-cli config generate

It is recommended to create secrets and default config with
mas-cli config generate > config.yaml (docs).
This works with shell, bash or similar tools and docker run.
In a kubernates only deployment it is not possible to create the config file.
Create the config file (> config.yaml) is part of the functional scope of a shell. The distroless container has no shell and cannot create the file.
IMO  it would be good if mas-cli wrote the file itself.

That makes sense! If one would implement such feature, they would:

add a flag on the config dump/generate commands here (here is in an example of option where a file is being passed)
open that file and write to it here instead of stdout, if provided

If you need a shell in the container, please note that there is a -debug variant that has the busybox tools installed,


Add update syntax and some reserved words

What type of PR is this：

[ ] bug
[x] feature
[ ] enhancement
[ ] others

Which issues of this PR fixes ：

Fixes #3932
Problem Summary(Required) ：
This PR adds update syntax, so users can update primary key tables using SQL. It adds update-related syntax parser, analyzer, and planer.

Only non-key columns are allowed to be updated
To prevent misuse user must specify a where clause to prevent a full table update.
Full row update is used for now, because after some reconsideration, write only partial update is incomplete to support update where xxx syntax
A read-write transaction style(select then full row update) approach is used currently, this won't guarantee transaction isolation(need table lock mechanism, will design&implement in future), but still it's useful, use this with this limitation in mind, do not perform updates with other concurrent loads/updates.

[FE PR Coverage check]
:disappointed: fail : 93 / 185 (50.27%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
0
2
00.00%

:large_blue_circle:
com/starrocks/sql/UpdatePlanner.java
0
36
00.00%

:large_blue_circle:
com/starrocks/sql/StatementPlanner.java
0
6
00.00%

:large_blue_circle:
com/starrocks/sql/plan/ExecPlan.java
0
8
00.00%

:large_blue_circle:
com/starrocks/sql/ast/AstVisitor.java
0
1
00.00%

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
2
25
08.00%

:large_blue_circle:
com/starrocks/analysis/DmlStmt.java
1
2
50.00%

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
15
23
65.22%

:large_blue_circle:
com/starrocks/sql/ast/ColumnAssignment.java
6
7
85.71%

:large_blue_circle:
com/starrocks/analysis/UpdateStmt.java
13
15
86.67%

:large_blue_circle:
com/starrocks/sql/analyzer/UpdateAnalyzer.java
54
58
93.10%

:large_blue_circle:
com/starrocks/sql/analyzer/Analyzer.java
2
2
100.00%

run starrocks_fe_unittest

[FE PR Coverage check]
:disappointed: fail : 118 / 156 (75.64%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/sql/plan/ExecPlan.java
0
8
00.00%

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
2
25
08.00%

:large_blue_circle:
com/starrocks/analysis/DmlStmt.java
1
2
50.00%

:large_blue_circle:
com/starrocks/sql/UpdatePlanner.java
33
36
91.67%

:large_blue_circle:
com/starrocks/sql/analyzer/UpdateAnalyzer.java
29
31
93.55%

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
21
22
95.45%

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/analyzer/Analyzer.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/StatementPlanner.java
6
6
100.00%

:large_blue_circle:
com/starrocks/sql/ast/AstVisitor.java
1
1
100.00%

:large_blue_circle:
com/starrocks/analysis/UpdateStmt.java
15
15
100.00%

:large_blue_circle:
com/starrocks/sql/ast/ColumnAssignment.java
6
6
100.00%

[FE PR Coverage check]
:heart_eyes: pass : 133 / 156 (85.26%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
7
25
28.00%

:large_blue_circle:
com/starrocks/sql/UpdatePlanner.java
33
36
91.67%

:large_blue_circle:
com/starrocks/sql/analyzer/UpdateAnalyzer.java
29
31
93.55%

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/analyzer/Analyzer.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
22
22
100.00%

:large_blue_circle:
com/starrocks/sql/StatementPlanner.java
6
6
100.00%

:large_blue_circle:
com/starrocks/sql/plan/ExecPlan.java
8
8
100.00%

:large_blue_circle:
com/starrocks/sql/ast/AstVisitor.java
1
1
100.00%

:large_blue_circle:
com/starrocks/analysis/UpdateStmt.java
15
15
100.00%

:large_blue_circle:
com/starrocks/analysis/DmlStmt.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/ast/ColumnAssignment.java
6
6
100.00%

run starrocks_fe_unittest

run starrocks_fe_unittest

[FE PR Coverage check]
:heart_eyes: pass : 133 / 156 (85.26%)
file detail

path
covered line
new line
coverage

:large_blue_circle:
com/starrocks/qe/StmtExecutor.java
7
25
28.00%

:large_blue_circle:
com/starrocks/sql/UpdatePlanner.java
33
36
91.67%

:large_blue_circle:
com/starrocks/sql/analyzer/UpdateAnalyzer.java
29
31
93.55%

:large_blue_circle:
com/starrocks/sql/analyzer/AnalyzerUtils.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/analyzer/Analyzer.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/parser/AstBuilder.java
22
22
100.00%

:large_blue_circle:
com/starrocks/sql/StatementPlanner.java
6
6
100.00%

:large_blue_circle:
com/starrocks/sql/plan/ExecPlan.java
8
8
100.00%

:large_blue_circle:
com/starrocks/sql/ast/AstVisitor.java
1
1
100.00%

:large_blue_circle:
com/starrocks/analysis/UpdateStmt.java
15
15
100.00%

:large_blue_circle:
com/starrocks/analysis/DmlStmt.java
2
2
100.00%

:large_blue_circle:
com/starrocks/sql/ast/ColumnAssignment.java
6
6
100.00%


Fix canvas stacking not deterministic on same layer

Fixes #22687.
If this is approved, I'll submit a back-compat PR for 3.0, so this is not suited for cherry-picking.
However, the compatibility breakage is minimal. Should only affect those dealing directly with the VisualServer. For CanvasLayer end-users nothing changes, but the behavior being deterministic.

@akien-mga I am fine with this, it seems to solve the problem elegantly

Also I dont think any project should be affected by this

Let's merge then. Thanks for the fix @RandomShaper!

Thank you!


Skip bigtableIO write error handling test on Dataflow runner

Mitigate #32047, leave only one error handling test for 5 currently exercised on Dataflow runner that experiencing same flaky behavior.
These tests the DoFn error handling of bad element, generally sufficient to be exercised on direct runner test suite. Dataflow suites already take very long to finish.
Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

assign to next reviewer


Feature/remove the words from the logo

Description
Per Brian's request, the words from the logo have been removed. I edited the original logo file, created and saved another called "HRF_Logo2". I left the original in tact should we need to revisit.
Video
Fixes # (issue)
Cosmetic issue.
Type of change
Please delete options that are not relevant.

[?] Bug fix (non-breaking change which fixes an issue)
[X] Stakeholder request.

Checklist:

[X] My code follows the style guidelines of this project
[X] I have performed a self-review of my own code
[X] I have removed unnecessary comments/console logs from my code
[X] I have made corresponding changes to the documentation if necessary (optional)
[X] My changes generate no new warnings
[X] I have checked my code and corrected any misspellings
[X] No duplicate code left within changed files
[X] Size of pull request kept to a minimum
[X] Pull request description clearly describes changes made & motivations for said changes

Why did your package-lock.json have so many changes?

Why did your package-lock.json have so many changes?

I'm not quite sure. Ryan shared one yesterday that had like 13k lines missing, so I wonder if this is similar.


Generated functionality

When the files and directories have been created, then I can see the styles are linked, as well as the JS functions to handle the anchor links within.

Still require valid links with the projects anchor tags, this is a part of mvp so needs addressing asap


Final countdown

Acrolinx Scorecards
A minimum Acrolinx score of 80 is required.
Click the scorecard links for each article to review the Acrolinx feedback on grammar, spelling, punctuation, writing style, and terminology:

Article
AcrolinxScore
LowestCategoryScore
Issues
Scorecard
Processed

enterprise-graph/conflation-model-tutorial.md
89
71
18
link
:white_check_mark:

enterprise-graph/graph-queries-tutorial.md
91
59
6
link
:white_check_mark:

enterprise-graph/ontology-tutorial.md
81
53
17
link
:white_check_mark:

enterprise-graph/quickstart.md
75
57
39
link
:white_check_mark:

enterprise-graph/schema-map-tutorial.md
77
28
3
link
:white_check_mark:

enterprise-graph/source-schema-concepts.md
75
41
25
link
:white_check_mark:

enterprise-graph/source-schema-tutorial.md
82
39
10
link
:white_check_mark:

enterprise-graph/toc.yml
100
100
0
link
:white_check_mark:

More info about Acrolinx

Install Acrolinx locally for VSCode
False positives or issues

OPS Build status updates of commit 9db7315:
:warning: Validation status: warnings

File
Status
Preview URL
Details

enterprise-graph/conflation-model-tutorial.md
:warning:Warning

Details

enterprise-graph/graph-queries-tutorial.md
:warning:Warning

Details

enterprise-graph/ontology-tutorial.md
:warning:Warning

Details

enterprise-graph/source-schema-tutorial.md
:warning:Warning

Details

enterprise-graph/conflation-validation-tutorial.md
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/13-cities-schema-uploaded.png
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/14-sales-customers.png
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/add-source-schema.png
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/all-schema-uploaded.png
:white_check_mark:Succeeded

enterprise-graph/quickstart.md
:white_check_mark:Succeeded

enterprise-graph/schema-map-tutorial.md
:white_check_mark:Succeeded

enterprise-graph/source-schema-concepts.md
:white_check_mark:Succeeded

enterprise-graph/toc.yml
:white_check_mark:Succeeded

enterprise-graph/conflation-model-tutorial.md

Line 75: [Warning] Invalid file link:(~/media/conflation-tutorial/C12-expanded-rule-set.png).

enterprise-graph/graph-queries-tutorial.md

Line 44: [Warning] Invalid file link:(~/media/graph-queries-tutorial/entity-details-view.png).

enterprise-graph/ontology-tutorial.md

[Warning] Author stflanag cannot be recognized in GitHub.

enterprise-graph/source-schema-tutorial.md

[Warning] Author stflanag cannot be recognized in GitHub.
Line 36: [Warning] Invalid file link:(~/schema-map-conepts.md).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

OPS Build status updates of commit 9db7315:
:warning: Validation status: warnings

File
Status
Preview URL
Details

enterprise-graph/conflation-model-tutorial.md
:warning:Warning

Details

enterprise-graph/graph-queries-tutorial.md
:warning:Warning

Details

enterprise-graph/ontology-tutorial.md
:warning:Warning

Details

enterprise-graph/source-schema-tutorial.md
:warning:Warning

Details

enterprise-graph/conflation-validation-tutorial.md
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/13-cities-schema-uploaded.png
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/14-sales-customers.png
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/add-source-schema.png
:white_check_mark:Succeeded

enterprise-graph/media/schema-maps-concepts/all-schema-uploaded.png
:white_check_mark:Succeeded

enterprise-graph/quickstart.md
:white_check_mark:Succeeded

enterprise-graph/schema-map-tutorial.md
:white_check_mark:Succeeded

enterprise-graph/source-schema-concepts.md
:white_check_mark:Succeeded

enterprise-graph/toc.yml
:white_check_mark:Succeeded

enterprise-graph/conflation-model-tutorial.md

Line 75: [Warning] Invalid file link:(~/media/conflation-tutorial/C12-expanded-rule-set.png).

enterprise-graph/graph-queries-tutorial.md

Line 44: [Warning] Invalid file link:(~/media/graph-queries-tutorial/entity-details-view.png).

enterprise-graph/ontology-tutorial.md

[Warning] Author stflanag cannot be recognized in GitHub.

enterprise-graph/source-schema-tutorial.md

[Warning] Author stflanag cannot be recognized in GitHub.
Line 36: [Warning] Invalid file link:(~/schema-map-conepts.md).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Fix low gas price to 0.0025 in osmosis

The v15 upgrade implements Osmosis Proposal 354, which sets a minimum gas price of 0.0025 uOSMO at the protocol-level.
https://github.com/cosmos/chain-registry/blob/master/osmosis/chain.json#L22
source
https://medium.com/osmosis-community-updates/osmosis-v15-0-0-sodium-upgrade-abb02767283d

@Rodrigoberes24 added a PR, thanks!


i18n(ko-KR): create legacy-flags.mdx

Description (required)

create legacy-flags.mdx
#9240

Related issues & labels (optional)

Closes #
Suggested label:

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

File
Note

ko/reference/legacy-flags.mdx
Localization added, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


Is this package still in ongoing development?

I really like the philosophy and design of this package but it appears its development process has basically stopped. I don't mean to pressure or to be rude but if @minimaxir could share his thoughts on this I would really appreciate it.

Can we expect development by the maintainer?
How can the community support?
Are you open to PRs and reviewing them?
What's your time commitment?
Should someone just fork the repo and develop it from there?

Thanks

To be perfectly clear, this is absolutely pressure and being rude. My day-to-day job that actually pays me a salary wouldn't ever provide a bulleted list of the reasons I suck, let alone a project I develop in my spare time.
As mentioned in #91, I had a mental breakdown which is why I had to take a break and am just returning to work on it now. The reason I took a break is because there's nothing broken with simpleaichat currently, nor are there any urgent issues. I intentionally designed simpleaichat to have minimal dependencies and scope to avoid those issues and that has worked so far (OpenAI's demo day this week affected a bit but I'll add a few Issues for that).
Large open source AI organizations like Hugging Face and LangChain have created a false sense that open-source AI projects have to always be shipping 🚀🚀🚀. The difference is that both of those are maintained by those who do it as their full-time job and the companies are backed by significant venture capital. If I could devote 100% of my professional time to simpleaichat and other open-source projects, I would, but I do need a salary.
You also don't need to ask for permission for forking if you think it is necessary. That's the beauty of open source with permissive licensing. If a fork is better than what is the case now due to my inactivity, that's a good thing for everyone!
tl;dr as the MIT License that this repo follows says:

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND

Hey Max, sorry to hear about your mental breakdown; I hope things are going better now!
You clearly created a library that hits a nerve with many developers; 3k stars is pretty impressive!
The misunderstanding here is (I think) that people using this library run into issues and would like them solved and are (most of the time) happy to create a PR to resolve it. My impression is that you prefer to keep control of the code (please correct me if I'm wrong). If that is indeed the case, just say so in the README. That sets the expectations for this project straight and will offload any stress this might cause you.
Just my 2c <3

Everyone seems to think that there's such a thing as GitHub Issues-zero or PR-zero, which is an impossible standard. Every nontrivial open source project will have a queue, which necessitates a triage priority. Not all issues are equal.
I have accepted and merged PRs which have fixed issues / errors. That is not the case with every PR or open issue, e.g. the open ones regarding support to add Azure OpenAI support which is not as important of a feature and require significant QA work. There is a very big difference between important and urgent. I've done emergency fixes (and taken vacation time for it) for issues such as the Pydantic 2.0 migration, because that indeed caused breaking issues.
Again, unlike Hugging Face and LangChain repos, this project is very intentionally limited in scope to decrease the needed effort to maintain it, and I have refused to make additions which will complicate it.
I have responded to questions in GitHub issues with suggestions / ideas for a PR, and that's worked fine. With any open source project, there's always the possibility that an issue won't be followed-up on or a PR gets rejected: that's just the industry.

Sorry, to hear about your situatio, Max. I really tried to be as soft and nice in my initial message while still getting my question across. I even asked how the community can support. I still think these are fair things to ask especially if I see a patreon link and sponsor button.
Anyway, I hear what you said and I disagree with many of them. You built a great library but I think it doesn't align with a project I want to support at the moment.
All the best to you and your projects.

Hi @pietz , I apologize for my emotional response to your question.
I've clarified my stance in #95 , which is materially similar to what I posted here so will likely not change your opinion.