Add lists to embedding metadata

Description of changes

Improvements & Bug fixes

closes: #227

New functionality

Can add lists of string, ints, or floats to embedding metadata
Can use $contains operator on lists in where filters

Examples:

# Adding lists
collection.add(
    embeddings=[[1.0, 1.1], [2.0, 2.2]],
    ids=["id_1", "id_2"],
    metadatas=[
        {
            "strings": ["a", "b", "c"],
            "ints": [1, 2, 3],
            "floats": [1.0, 2.0, 3.0],
        },
        {
            "strings": ["d", "e", "f"],
            "ints": [4, 5, 6],
            "floats": [4.0, 5.0, 6.0],
        },
    ],
)

# Updating lists
collection.update(
    ids=["id_1", "id_2"],
    metadatas=[
        {
            "strings": ["a_1", "b_1", "c_1"],
        },
        {
            "strings": ["d_2", "e_2", "f_2"],
            "ints": [9, 10],
        },
    ],
)

# where filters
items = collection.get(where={"strings": {"$contains": "a_1"}})
items = collection.get(
    where={"$or": [{"strings": {"$contains": "a_1"}}, {"ints": {"$contains": 9}}]}
)

Test plan
Existing tests + new tests passing locally
Documentation Changes
Docs will need an update describing metadata and where filters

LGTM 🚀

Update on the way to address conflicts
I think I should also add property/hypothesis tests?
Just want to confirm that my solution here is directionally correct. @HammadB ?

:robot: :rocket: LGTM 🚀 🤖

Fixed the failing test, was a result of a bad merge
Working on adding hypothesis strategies to property tests for adding and filtering metadata lists. Will push this week

LGTM

Totally forgot about tests for the JS client. A couple of tests expected an error when using $contains in ` a where filter. Just pushed a fix

Have some updated property tests for lists running (and passing) locally. Currently blocked by some mypy errors from the precommit hook

@HammadB can you try running the tests again?

Hi @Russell-Pollari , thanks so much for the PR. We definitely want to implement this feature so appreciate the contribution.
And, thanks for working on the property tests. Having those rock solid with this new feature are definitely necessary before we merge, so we can make sure we've handled edge cases and have feature parity across all implementations. Much appreciated.
I have one higher-level question before reviewing this code in detail. The PR talks about adding "list" support, but in your schema, I don't see any notion of ordering at the storage layer. Which means we're dealing with logical sets, rather than lists. I think that's ok. Sets fully allow us to answer contains? queries, and then we don't have to worry about the complexities of list ordering.
But in that case, I'd want to make sure that the semantics are precise and tested:

The code and docs should say "set" instead of list consistently.
eq and neq should follow set semantics and ignore order.
Adding an item to a set where it's already present should be a no-op. (And, specifically, if I "add" an item multiple times then "remove" it once, the set should not end up containing the item.)
We should return appropriate set data structures in the relevant languages wherever possible (accepting lists as inputs is fine though.)

Does that make sense?

@levand

Which means we're dealing with logical sets, rather than lists.

Good point! Should have made the same conclusion after adding a _metadata_lists_to_sets helper function in the tests. :disappointed:
If we're talking sets instead of lists, do you think the current schema/storage layer is a good approach?
Meanwhile, I'll get to work on your suggestions

The code and docs should say "set" instead of list consistently.

Roger that! Can refactor to sets

eq and neq should follow set semantics and ignore order.

My current implementation only allows $contains operator for filtering by metadata ~lists~ sets. Should I also support $eq and $ne`?

Adding an item to a set where it's already present should be a no-op. (And, specifically, if I "add" an item multiple times then "remove" it once, the set should not end up containing the item.)

Agreed. This will require a bit of reworking, but do-able. Will get to it.

We should return appropriate set data structures in the relevant languages wherever possible (accepting lists as inputs is fine though.)

Roger that!

We should return appropriate set data structures in the relevant languages wherever possible (accepting lists as inputs is fine though.)

My first pass here is to require different types — one for end users, allowing them to input lists. And another for the storage layer to require logical sets.
Metadata and  UpdateMetadata seem to serve a similar purpose already. Here's how I would modify them. Should probably rename the latter to something like AddOrUpdateMetadata
# Metadata lists are stored as sets, but we allow users to provide lists for convenience
MetadataValues = Union[str, int, float, bool]
Metadata = Mapping[str, Union[MetadataValues, Set[MetadataValues]]]
UpdateMetadata = Mapping[
    str, Union[MetadataValues, Set[MetadataValues], List[MetadataValues], None]
]

@Russell-Pollari Awesome, thanks for the responsiveness!

If we're talking sets instead of lists, do you think the current schema/storage layer is a good approach?

I think it's ok. To make it a set (preventing duplicates) instead of a bag (duplicates allowed), maybe include the value in the primary key? Then set uniqueness could be enforced at the database level.
The other option would be to use a json data structure or something like that for the set value, as a column instead of its own table. But that probably wouldn't support fast contains? queries without special secondary json indexes and I can't recall if sqlite supports those or not.
A table is probably the right answer.

My current implementation only allows $contains operator for filtering by metadata lists sets. Should I also support $eq and $ne`?

I think making contains? the only supported operator for sets is ok. I can't imagine a use case for set (in)equality, and we can add it later if necessary. @HammadB what do you think?
Thanks!

@levand
After some prototyping, I think it would be better to stick to lists and update the schema to preserve list order. This might make the storage layer a tad more complex, but it saves some complexity on the APIs.
Since sets are not JSON serializable, there has to be a lot of conversion from lists to sets and vice-versa. The result, IMO, is a worse UX for developers using the API
On the lists front, we can preserve order by adding the list index to the table
CREATE TABLE embedding_metadata_lists (
    id INTEGER REFERENCES embeddings(id),
    key TEXT NOT NULL,
    list_index INTEGER NOT NULL,
    string_value TEXT,
    float_value REAL,
    int_value INTEGER,
    bool_value BOOLEAN,
    PRIMARY KEY (id, key, list_index),
    FOREIGN KEY (id, key) REFERENCES embedding_metadata(id, key) ON DELETE CASCADE
);

What do you think?

@Russell-Pollari this is great work!
This might be a bit of a tangent question, but if we'd need to query where an value is not in a metadata array property, would this still work? Happy to provide more details for usecases, but thought i might pop the question since there currently is support for $ne (in the context of single values).

@andrewmurraydavid
This PR only introduces $contains for lists. But would be great to follow up with more operators for lists (e.g. $not_contains, $all)

@Russell-Pollari got it. Im trying to understand why this PR wouldn't add $not_contains functionality since that would create consistency between single values and array values operations.
Not trying to be pushy, just curious.

@andrewmurraydavid There would be some uphill work involved with other operators, and I don't want to add more complexity. It's best to keep PRs to change one thing at a time.
This PR is already sizeable and still needs review by the Chroma team. But if they approve of my approach, I will definitely get to work on more operators

@Russell-Pollari

Since sets are not JSON serializable, there has to be a lot of conversion from lists to sets and vice-versa. The result, IMO, is a worse UX for developers using the API

I think it's ok to use a list in json if we document and treat it like a set (never do anything that depends on order, and assume it has no duplicates.) Developers shouldn't need to be exposed to the conversion, right? And they can still pass in lists (we'll handle the conversion for them, we just need to document that dups will be removed and order won't be maintained.)
I might be missing something though... how do you think this would affect the UX for developers? Are there specific operations that you think would be confusing or ambiguous?
Your approach to adding the list index to the table will certainly work (albeit at the cost of backend complexity/performance)... curious to think through the UX implications more thoroughly before we make that decision.

@levand I'm probably overstating the UX issue. But wouldn't this would require maintaining separate types for user input and return values? Sending a List and getting a Set in return.
I suppose good documentation can get around this, but sticking to lists would let developers grok all they need just from looking at the Metadata type.
From a code perspective, I also think sticking with lists would be less complex overall (as the changes touch considerably less parts of the code base). Though I don't have a good sense of the relative performance costs of each.

@Russell-Pollari ok. I'm going to defer to @HammadB on the DevX issue since he has a way better intuition than I do about what our users need/want.
I can live with with either sets or lists provided we are consistent with the semantics of whichever we choose.

I agree with @levand that we should preserve the semantics of what we choose (I.E If we do lists, they should preserve order).
I am not convinced the overhead of users expecting list semantics is worth the implementation cost (however the cost itself is unclear to me). What are the use cases where order matters?
The main use case I think we want to support is I have a bag-of-ints/floats/strings and users want to check if a value is in the target bag.
What I'd like to see is an analysis of the set impl vs the list impl from a performance and schema perspective. Also I need to think about the distributed case.

What are the use cases where order matters?

Honestly, I can't think of any. But my thinking, which may be wrong, is that sticking with Lists simplifies the implementation and the required mental models for users while still solving the main use case—which I agree is to check if a value is in the target bag/list.

What I'd like to see is an analysis of the set impl vs the list impl from a performance, use-case and schema perspective. Also I need to think about the distributed case.

Roger that. I have the lists impl 95% complete. Will confirm all tests pass locally and push to this branch and mark it as ready to review. Will also try and push up another branch with a sets implementation.
Do you have a recommended method(s) to test performance? @HammadB @levand

@HammadB, a bit of a tangential question related to this. Have you considered using  JSON1 Extensions (delivered by default with most SQLite distros since 3.9).
The distinct benefits are:

Arbitrary objects as metadata
JSONPath expressions
Will cover a more extensive set of requirements from end-users with a single change
Less technical debt when it comes to multiple implementations for various set/list/dict etc. semantics
Potentially extend the supported datatypes without further implementation e.g. dates and timestamps (will need further consideration about the possible use of operators in Where)

Maybe there are things I am missing at this late hour.

@tazarov
I think the issue with that approach is that, with arbitrary JSON objects, you lose the ability to create indexes to speed up queries.

@tazarov I think the issue with that approach is that, with arbitrary JSON objects, you lose the ability to create indexes to speed up queries.

@Russell-Pollari, you might be right, and I don't think most people will be needing indexes over arbitrary data.
Still, SQLite seems to offer a way to create indexes based on JSON data:
CREATE INDEX idx_json_key ON <tbl> (json_extract(<col>, '$.key'));

Perhaps JSON indexes are something that can be added to Chroma further down the road.

@levand  @Russell-Pollari Any update on that?

What is the status on the implementation?

Have not had time to explore an alternative implementation with sets.
Will aim to get this branch up to date with main tomorrow

Any updates on this?  Definitely a very useful feature.

Anything we can do to get this PR merged?


Long virtual text not displayed correctly when wrap is disabled, and cursor is not displayed at the correct position

Steps to reproduce

Run vim --clean test.vim -S test.vim, with

vim9script

colorscheme habamax
set nowrap
prop_type_add('test', {highlight: 'Error'})

const opts: dict<any> = {
    type: 'test',
    text: range(50)->join(' '),
    text_align: 'after',
    text_padding_left: 2
}

prop_add(1, 0, opts)
prop_add(1, 0, opts)
prop_add(1, 0, opts)

Move the cursor to the bottom of the buffer by pressing G.
Result: The cursor does not jump to the last line.

Screenshot:

Furthermore, the 2nd and 3rd virtual lines are displayed below the first line even though text_align is set to after.
At least the issue with the cursor disappears when wrap is enabled.
Expected behaviour

Cursor should always be drawn at the right position.
When there are multiple virtual text-properties with text_align set to after, they should always be drawn one after another in one line like in the following screenshot:

Version of Vim
9.0.706
Environment
Linux
Logs and stack traces
No response

Steps to reproduce

Run vim --clean test.vim -S test.vim, with

vim9script

colorscheme habamax
set nowrap
prop_type_add('test', {highlight: 'Error'})

const opts: dict<any> = {
    type: 'test',
    text: range(50)->join(' '),
    text_align: 'after',
    text_padding_left: 2
}

prop_add(1, 0, opts)
prop_add(1, 0, opts)
prop_add(1, 0, opts)

Move the cursor to the bottom of the buffer by pressing G.
Result: The cursor does not jump to the last line.

Screenshot:

Furthermore, the 2nd and 3rd virtual lines are displayed below the first line even though text_align is set to after.
At least the issue with the cursor disappears when wrap is enabled.

This turns out to be a complicated problem.  Let me first fix that the
"after" properties are displayed on a separate line.  The cursor
position should also be fixed then.
However, when moving the cursor further right, causing the text to
shift left, the "after" property always shows, not skipping the start.
I'll tackle that later.
--
GUARD #2:  It could be carried by an African swallow!
GUARD #1:  Oh, yeah, an African swallow maybe, but not a European swallow,
that's my point.
GUARD #2:  Oh, yeah, I agree with that...
The Quest for the Holy Grail (Monty Python)
/// Bram Moolenaar -- @.*** -- http://www.Moolenaar.net   \
///                                                                      \
\\        sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ ///
\\            help me help AIDS victims -- http://ICCF-Holland.org    ///


Issue connecting via API

Hi,
I was very excited to use your work, however, the API call does not seem to be working.
I get the following error : ConnectionError: HTTPConnectionPool(host='<IP_ADDRESS>', port=8110): Max retries exceeded with url: /api/fh (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe5da1635e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Here's the code I am using :
def query_chemfh(df: pd.DataFrame):
	url = "http://<IP_ADDRESS>:8110/api/fh"
	n = 2000

	param = {'SMILES': []}
	smiles_list = df['SMILES'].tolist()
	data = {"SMILES": smiles_list}

	for _, sublist in enumerate(divide_list(smiles_list, n)):
		param['SMILES'] = sublist

		response = requests.post(url, json=param)

		if response.status_code == 200:
			data = response.json()['data']
			result = transform(data)
			return result

I tried pinging the server and that seems to work.
Using curl :
curl -X POST http://<IP_ADDRESS>:8110/api/admet -H "Content-Type: application/json" -d '{"SMILES": ["C"]}'
gives
curl: (7) Failed to connect to <IP_ADDRESS> port 8110 after 257 ms: Connection refused
If you could please advise on the best course of action, I'd be grateful

Dear Antoine,
Thank you for reaching out and bringing this issue to our attention.
We apologize for the inconvenience you are experiencing with the ChemFH webserver. Currently, we are in the process of maintaining and updating the server to enhance its performance. Unfortunately, our server is not equipped to handle a high volume of API requests at this time and has also been subjected to unidentified attacks, which have further impacted its stability.
We appreciate your understanding and patience while we work on these improvements. We will notify you as soon as the server is back up and running smoothly.
Kind regards,
Jiacai Yi
2024-06-06 05:01:30 "Antoine Lacour" @.***> 写道：
Hi,
I was very excited to use your work, however, the API call does not seem to be working.
I get the following error : ConnectionError: HTTPConnectionPool(host='<IP_ADDRESS>', port=8110): Max retries exceeded with url: /api/fh (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe5da1635e0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Here's the code I am using :
def query_chemfh(df: pd.DataFrame):
url = "http://<IP_ADDRESS>:8110/api/fh"
n = 2000
param = {'SMILES': []}
smiles_list = df['SMILES'].tolist()
data = {"SMILES": smiles_list}

for _, sublist in enumerate(divide_list(smiles_list, n)):
	param['SMILES'] = sublist

	response = requests.post(url, json=param)

	if response.status_code == 200:
		data = response.json()['data']
		result = transform(data)
		return result

I tried pinging the server and that seems to work.
Using curl :
curl -X POST http://<IP_ADDRESS>:8110/api/admet -H "Content-Type: application/json" -d '{"SMILES": ["C"]}'
gives
curl: (7) Failed to connect to <IP_ADDRESS> port 8110 after 257 ms: Connection refused
If you could please advise on the best course of action, I'd be grateful
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you are subscribed to this thread.Message ID: @.***>


Use liboverdrop

Supersedes: #7
Holding this while liboverdrop is packaged in Fedora.

Updated - with liboverdrop released and in Fedora, this is now ready for merge.

Updated!


Add sidekiq_report_type config option

Sidekiq takes 2 types of handlers to deal with failed jobs:

error handler - which will be called every time a job failed
death handler - which will only be called when the job is dead (no
retries remained)

And currently, our sidekiq integration is based on the error handler. So
if a job failed repeatedly, the exception will be sent multiple times.
There have been some requests on "only sending events when there's no
retry left" and I think this can be a solution for them.

Codecov Report

Merging #1022 into master will decrease coverage by 0.43%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1022      +/-   ##
==========================================
- Coverage   98.11%   97.68%   -0.44%
==========================================
  Files          57       58       +1
  Lines        2391     2415      +24
==========================================
+ Hits         2346     2359      +13
- Misses         45       56      +11

Impacted Files
Coverage Δ

lib/raven/integrations/sidekiq.rb
100.00% <100.00%> (ø)

lib/raven/integrations/sidekiq/death_handler.rb
100.00% <100.00%> (ø)

spec/raven/integrations/sidekiq_spec.rb
100.00% <100.00%> (ø)

...ations/rails/overrides/debug_exceptions_catcher.rb
73.33% <0.00%> (-26.67%)
:arrow_down:

...s/rails/overrides/debug_exceptions_catcher_spec.rb
88.88% <0.00%> (-11.12%)
:arrow_down:

...ven/breadcrumbs/active_support_breadcrumbs_spec.rb
94.44% <0.00%> (-5.56%)
:arrow_down:

lib/raven/integrations/rails/active_job.rb
96.42% <0.00%> (-3.58%)
:arrow_down:

...c/raven/integrations/sidekiq/error_handler_spec.rb
96.77% <0.00%> (-3.23%)
:arrow_down:

lib/raven/breadcrumbs/sentry_logger.rb
83.33% <0.00%> (-0.88%)
:arrow_down:

lib/raven/integrations/rails.rb
90.69% <0.00%> (-0.22%)
:arrow_down:

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff8ef67...19de77d. Read the comment docs.


updating to k8s 1.30 and other supporting dependencies to latest

Relates: #1163

/test 4.15-e2e


Support view commits by Hash ID on log graph window

Sensible request, one second, I'll have to lightly rewrite the Graph syntax to make it fit together.

@kemayo I have implemented it few days ago and will make pull request later.
Could you reopen this issue?

Do I need to? I mean, I already implemented what I think you're asking for...?

Do you mean the commit cf56cde327a552c46811fcd98af22aa8c39231bf ?
I suppose to Press a key to show commit when current cursor stay on Hash ID.
Just leave this issue for few days. I will update the details on description.
thanks

No, I mean 47a3228 -- the one that I closed the task with. It extends the existing "hit enter to go to commit in blame" behavior to also work in graph.

But I hit enter key and then nothing happens?
What is wrong?

Hard to say. Assuming there are no errors in the log, it's either that you're pressing enter on a line which doesn't contain a commit hash, or you've not updated to the version which includes the new functionality (I haven't tagged a release since making that change, for one thing).

Anyway, I will make pull request later and you will see the details.

Sure, I will be happy to examine it.


Bug: Deleting the only record in a cluster adds an extra "},"

The OST_ERASE_RECORD() proc incorrectly formats the cluster after the record is deleted. This needs to be fixed

Fixed in: 4fedad7c402cb642f12b7acae8367111276ba66f


Update faq.yml

New FAQ ASL AO

Description

As already mentioned in #ISSUE_NUMBER, this PR tackles:

...
...
...

In particular, the ...
Checklist

[ ] I followed the indications in the CONTRIBUTING
[ ] The documentation related to the proposed change has been updated accordingly (also comments in code).
[ ] Have you written new tests for your core changes, as applicable?
[ ] Have you successfully ran tests with your changes locally?
[ ] Ready for review! :rocket:

Fixes

Fixes #

please @bfabio @denismarini

It's done @dagdome!


RDMO 2.0.0

This PR fixes compatibility issues with RDMO 2.0.0.

Should this be merged now, that RDMO 2.0 is released, @jochenklar?


Show complete Org Config Info to Org Admins

Org Admins should not be able to edit any of their config info but we should also expose the Shib details in the read only panel. Screenshot 1 is the Super Admin view.
Add read only fields for 'Shibboleth Entity ID' and 'Shibboleth Domain' to the Org Admin view in Screenshot 2.

on second thought this is not a high priority. closing.


Inverted avatars :(

It seems to work fine on my end. I have tried creating a sample event in the "Month" view for today's date, but the avatars look fine.

I am on Firefox.


RFM - cura-lulzbot

Current Maintainer

[x] I am the maintainer of the package and wish to pass it to someone else;

Checklist

[x] Issue title starts with 'RFM - '

Existing Package Details
Package URL: https://chocolatey.org/packages/cura-lulzbot
Package source URL: https://github.com/jtcmedia/chocolatey-packages/tree/master/cura-lulzbot

/confirm jtcmedia

Package automated, this issue can be closed

@jtcmedia can yyou close this issue?


Add module-info.java

So that projects depend on this can be published to a public artifact repository.
Note that this is not breaking backward compatibility. All codes except this file can be still compiled in Java 6.

I don't see how publishing to a public artifact repository and becoming a
module jar are correlated.
On Wed, Oct 18, 2017, 3:52 AM hrchu<EMAIL_ADDRESS>wrote:

So that projects depend on this can be published to a public artifact
repository.
Note that this is not breaking backward compatibility. All codes except
this file can be still compiled in Java 6.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/square/okhttp/issues/3655, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEEEb54bHoM8K9PghQW85XsZwBBlXnuks5sta4zgaJpZM4P9TIu
.

Does seem like something we should do.
http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html
Along with additionally building against released JDK 9 in https://github.com/square/okhttp/blob/master/.travis.yml

Oh I'm all for an automatic module name. But I see no reason to add a
module-info.class entry.
On Wed, Oct 18, 2017 at 11:55 AM Yuri Schimke<EMAIL_ADDRESS>wrote:

Does seem like something we should do.
http://blog.joda.org/2017/05/java-se-9-jpms-automatic-modules.html
Along with additionally building against released JDK 9 in
https://github.com/square/okhttp/blob/master/.travis.yml
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/square/okhttp/issues/3655#issuecomment-337638759, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEEETwucSMzypw1PsuVxu4CFuRy0IeHks5sth9WgaJpZM4P9TIu
.

@JakeWharton the problem is that projects depends on filename-based automodules should not be published to a public artifact repository.
At least we need add Automatic-Module-Name entry to JAR's manifest.
https://github.com/google/guava/commit/f2be0be06ef49dee7c5dfcb3a0231c5ecaf5b621#diff-e407b6f223ee6c04116a91ead9f5e100R42
Refer: https://github.com/grpc/grpc-java/issues/3523

That's what I said, yeah. But why are you recommending a module-info.class
entry?
On Wed, Oct 18, 2017 at 9:53 PM hrchu<EMAIL_ADDRESS>wrote:

@JakeWharton https://github.com/jakewharton the problem is that
projects depends on filename-based automodules should not be published to a
public artifact repository.
At least we need add Automatic-Module-Name entry to JAR's manifest.
google/guava@f2be0be#diff-e407b6f223ee6c04116a91ead9f5e100R42
https://github.com/google/guava/commit/f2be0be06ef49dee7c5dfcb3a0231c5ecaf5b621#diff-e407b6f223ee6c04116a91ead9f5e100R42
Refer: grpc/grpc-java#3523 https://github.com/grpc/grpc-java/issues/3523
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/square/okhttp/issues/3655#issuecomment-337776700, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAEEEYjzrk2ka7oDWE6bpg3GoB6_Dxdeks5stquOgaJpZM4P9TIu
.

I change my mind.  Automatic-Module-Name is good enough currently.

https://github.com/square/okhttp/pull/3743

Per https://github.com/square/okhttp/pull/3743#issuecomment-363851149
We should (ideally) clean up package names to not overlap when doing this.
cc @jodastephen

Done.

Newbie here. Can't create a custom JRE using jlink without module-info.java. I use OkHTTP in almost all my projects. Would be great if I didn't have to manually unpack and recompile everything with module-info.java.


[Gally]: master <- dev

Automatically created by Git-Ally

:tada: This PR is included in version 1.10.119 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Add support for collect_logs in the cli

Allow users to quickly get a snapshot of the logs on the site. We use this in the event of a downtime to try capture as much data to debug and isolate issues.
This would help users reduce downtime and root cause analysis when things go awry or even during the course of regular development

This is a feature not yet supported by the Pantheon Platform or web UI. Feature requests of that nature should best be proposed via Support Tickets.

Not going to expose the internal btool stuff. Closing this in favor of tracking log access on #232


希望能支持curl调用纯大模型回答时候的query纠错和改写功能

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description
请求后：curl -X POST http://<IP_ADDRESS>:5000/api/v1/awel/trigger/examples/simple_chat
-H "Content-Type: application/json" -d '{
"model": "chatglm2-6b",
"user_input": "北京又拿写好玩的地方"
}'
能够输出query：”北京又拿写好玩的地方 “      的纠错query：“北京有哪些好玩的地方”和三条相似query。
Use case
No response
Related issues
No response
Feature Priority
None
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Search before asking

[x]  I had searched in the issues and found no similar feature requirement.

Description
请求后：curl -X POST http://<IP_ADDRESS>:5000/api/v1/awel/trigger/examples/simple_chat -H "Content-Type: application/json" -d '{ "model": "chatglm2-6b", "user_input": "北京又拿写好玩的地方" }'
能够输出query：”北京又拿写好玩的地方 “ 的纠错query：“北京有哪些好玩的地方”和三条相似query。
Use case
No response
Related issues
No response
Feature Priority
None
Are you willing to submit PR?

[ ]  Yes I am willing to submit a PR!

这个后面版本会支持。

Search before asking

[x]   I had searched in the issues and found no similar feature requirement.

Description
请求后：curl -X POST http://<IP_ADDRESS>:5000/api/v1/awel/trigger/examples/simple_chat -H "Content-Type: application/json" -d '{ "model": "chatglm2-6b", "user_input": "北京又拿写好玩的地方" }'
能够输出query：”北京又拿写好玩的地方 “ 的纠错query：“北京有哪些好玩的地方”和三条相似query。
Use case
No response
Related issues
No response
Feature Priority
None
Are you willing to submit PR?

[ ]   Yes I am willing to submit a PR!

这个后面版本会支持。

想问现在有纠错功能了吗？执行sql会出问题


[Bug]: Inconsistent severity assigned to iam_no_custom_policy_permissive_role_assumption

Steps to Reproduce
This issue concerns the severity classification for check ID iam_no_custom_policy_permissive_role_assumption.
The check has critical severity. It flags policies that allow unconstrained sts:AssumeRole.
Why is this check classified as critical, especially when compared to other related checks, which are classified only as high? How can an attacker abuse unconstrained sts:AssumeRole to escalate privileges in a generic way?
I understand that sts:AssumeRole allows assuming any role in theory. However, the role's trust policy is also consulted. Roles can only be assumed if their trust policy explicitly allowlists the caller. In addition, the role's name might not be public and is not always known.
Other AWS IAM checks that are similar are

iam_policy_allows_privilege_escalation: classified as high, generically covering multiple options to escalate privileges
iam_inline_policy_no_administrative_privileges: classified as high, checks for administrative permissions assigned inline.

Expected behavior
I expect severity scores to be comparable across similarly-severe issue. However, I don't understand why the severities for the checks mentioned above differ. Part of this might be that the check's metadata does not sufficiently explain the risk, which is stated as: If not restricted unintended access could happen..
Actual Result with Screenshots or Logs
See above.
How did you install Prowler?
Cloning the repository from github.com (git clone)
Environment Resource
Not relevant / Workstation
OS used
Not relevant / macOS
Prowler version
4.1.0
Pip version
24.0
Context
Note: I've been writing up relatively minor points as issues on GitHub. I do this because I think it's useful for others to be able to revisit and find issues, and I believe GitHub's issue search to be more accessible than Slack. Let me know if you'd prefer I raise these points via chat.

Sounds good, I'll create a PR.

How can an attacker abuse unconstrained sts:AssumeRole to escalate privileges in a generic way?
Do you have further insights to this question? Would love to know if there is something I'm missing.

Sounds good, I'll create a PR.

Great, thanks!

How can an attacker abuse unconstrained sts:AssumeRole to escalate privileges in a generic way?

Do you have further insights to this question? Would love to know if there is something I'm missing.

The check iam_no_custom_policy_permissive_role_assumption aims to comply with the Least Privilege Principle enforcing not to allow policies to assume whatever role, but as you well mentioned the role assumption in AWS requires also to check the role's trust relationship. Even with that if Resource is set to *, the user can assume any role in any account that trusts the user's account. (In other words, the role's trust policy specifies the user's account as Principal). As a best practice, the official recommendation from AWS is to specify the complete ARN for only the roles that the user needs.
Also I'll add this https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_permissions-to-switch.html#roles-usingrole-createpolicy to the RelatedURL field in the check's metadata.


[idea] Setting to configure word regex.

For example, I am using this, and would like to match the regex that vim-mode uses for * and #. Possibly would be nice to add option to match vim-mode's regex, or to allow a custom regex.

Yeah, I have this idea from start, this package is port of my vim-quickhl to Atom.
Original vim version have regex option and static pattern(always highlighted keywords without manual operation).
It's good to have, but I have many thing to do in other project, so if someone sent PR I'll welcome to look into.
Or else, I can implement if I have time and want some refreshment.
Thanks for suggestion.


Add TfsPullRequestSettings overload which uses environment variables from Build

Add TfsPullRequestSettings overload which uses available environment variables from TFS / Azure DevOps build (BUILD_REPOSITORY_URI, SYSTEM_PULLREQUEST_PULLREQUESTID)

Could you please clarify how this new constructor should work?
As far as I understand:

it should only accept ITfsCredentials as a parameter
it should assert the value of repository Url it gets from the env for null
it should NOT assert either source branch or pull request ID values (as far as I can see, it is handled during the construction of the TfsPullRequest object)

Correct?

There are multiple things which can be done here:

Add an Ctor overload public TfsPullRequestSettings(ITfsCredentials credentials) which reads BUILD_REPOSITORY_URI and SYSTEM_PULLREQUEST_PULLREQUESTID environment variable, asserts that they are set and contain a value and the sets this.RepositoryUrl and this.PullRequestId.
This helps to construct a settings object in a Cake script which is executed in a TFS / Azure DevOps build.
Create a static method TfsPullRequestSettings.UsingTfsBuildOAuthToken() which returns an instance using the new ctro from the previous point by passing an instance of TfsAuthenticationOAuth to it with the value from the SYSTEM_ACCESSTOKEN environment variable.
This helps to construct a settings object in a Cake script which is executed in a TFS / Azure DevOps build which has an OAuth token provided to the build.
Add an alias TfsPullRequestUsingTfsBuildOAuthToken(this ICakeContext context) which uses settings created by the static method from the previous point.
This makes it even simpler to get access to the pull request in a TFS / Azure DevOps build which has an OAuth token provided to the build.


Redesign creature editor

The creature editor needs to be redesigned in two major ways

It needs to fully support joysticks
It needs to be nicer to look at. Look at creature editors for games like Minecraft and Starbound, it shouldn't have things like dropdown menus and fonts. It should use pictures and nice pretty buttons.

Additionally, we should get rid of the "import/export" functionality (or perhaps have this as an optional dev mode thing.)

Replaced with #2445.

Closing. The creature editor has been redesigned.


border Challenges

Issue Description

When on my phone coding with free code camp
Trying to code challenges concerning border, all of them, with the right code (same challenges with same code done on laptop worked, but not on the phone.

Browser Information
*Google App version <IP_ADDRESS>.arm64

Android version 5.1.1
Sony Xperia Z6

The right code for challenges concerning border
If relevant, paste all of your challenge code in here

Screenshot

@cienci can you provide any specific challenges and your code for us to test?


0.2 path fix

Defines a main_route_path and admin_route_path so that the route names and path names are independent.  This allows the ticket system to be placed inside of subpaths.  Issue #147
Service provider aborts boot if the database isn't installed.

Works like a charm, thank you very much :+1:


malwaredomainlist is still listed??

the list has been down for quite a while, at least 8 months or so I'd say (remember only cause it was "alive" when I first got my pi hole setup....)  - well now the list is completely gone and just shows <IP_ADDRESS> localhost when linked to that blocklist.

This has been removed!


Burn typology

This issue was originally posted on the googlecode site.  I received a response that this request is outside of the domain of OGMS, but I am looking for suggestions for which ontology to request these terms in.  I was directed here
original post:
We are representing 'burn patient role' in OMRSE.  The definition is, 'a patient role that inheres in an organism suffering a thermal, electrical, chemical or radiation burn.'
We are using OGMS 'injury' to represent a 'injured patient role', and we would like to use burns analogously.
We request the following hierarchy of terms under 'injury':
'burn'
'thermal burn'
'electrical burn'
'chemical burn'
'radiation burn'

I'd like to propose a definition for 'burn'; however, before doing, I'd like to offer a brief explanation of each burn subtype.

thermal burn - tissue damage caused by an increase in temperature beyond the normal limits of the body, in which the internal kinetic energy of a system is transferred (i.e., heat transfer) to an organism causing structural damage to proteins. This does not include burns caused by heat exposure from an electrical current. (Vehicle = molecules, atoms, and/or electrons)
electrical burn - tissue damage caused by exposure to electrical currents whose voltage, flow of electric charge, resistance, and/or frequency are beyond the normal limits of the body, which lead to structural damage to plasma membranes and the proteins therein. (Vehicle = electrons and/or other ions)
radiation burn - tissue damage caused by electromagnetic waves interacting with the physical components of tissue in a way that induces damaging chemical responses. (Vehicle = photons)
chemical burn - tissue damage caused by direct contact with a corrosive substance.

As you might see, each of these burns involves a supraphysiological change that is the result of a physical interaction between an organism and some instance of a material entity (e.g., an electron, photon, corrosive substance, etc.). This material entity is exogenous to the body and has some disposition to induce bodily harm. As such, my proposed definition for 'burn' is:
"An injury caused by some pathological transformation that has as participant some material entity which is the bearer of some disposition to induce bodily harm upon direct contact with the tissue of an organism."

I received a response that this request is outside of the domain of OGMS, but I am looking for suggestions for which ontology to request these terms in. I was directed here

I also agree that specific types of injuries don't belong in OGMS. Just as it isn't the place for typologies of diseases or syndromes, etc. But until some enterprising folks take the initiative to create an injury ontology, perhaps OGMS can act as an incubator for terms like these. I will propose it on the next developers call.

Closed ticket because issue has not been acted upon in reasonable amount of time.


Implement tests for ProvStore GET /store/api/v0/documents/:id(/flattened)(/view/:view)

I could not get REST invocations to work for:

GET /store/api/v0/documents/:id(/flattened)
GET /store/api/v0/documents/:id(/flattened)(/view/:view)

Hi @mikej888,
Could you give more information about the issues you got when trying the above requests?
They are still working at my end with the following examples:
curl -L "https://provenance.ecs.soton.ac.uk/store/api/v0/documents/147/" \
	-H "Accept: text/provenance-notation"

curl -L "https://provenance.ecs.soton.ac.uk/store/api/v0/documents/147/flattened/" \
	-H "Accept: text/provenance-notation"

curl -L "https://provenance.ecs.soton.ac.uk/store/api/v0/documents/147/flattened/views/data" \
	-H "Accept: text/provenance-notation"

Hi @trungdong,
Problem solved, thanks to seeing these! I wasn't setting the Accept type. I've added tests for these now: a832e1d74d3652e0e21be4e9d933975760a7cfac!


Fix row indentation on iPad

When using AHKActionSheet on an iPad, the rows get indented. This pull request fixes that indentation on the iPad.  See attached before and after screenshots.

Is this going to be merged eventually? I'm making an app and using this control on iPad and I have this indentation issue. Thanks!

Sorry for the delay. Merged now!


Help about the bert on Yahoo Answers.

I am trying to run vfl_framework.py based on the dataset Yahoo Answer. I have downloaded the Bert model 'bert-base-uncased' and saved it in models/transformers/. However, something is still wrong.

i download the bert to local and try to import with:
self.bert = BertModel4Mix.from_pretrained('./models/Bert/bert-base-uncased')
but its final top-1 accuray is ~37%
So it make me confused

i download the bert to local and try to import with: self.bert = BertModel4Mix.from_pretrained('./models/Bert/bert-base-uncased') but its final top-1 accuray is ~37% So it make me confused

Hello, I have solved the problem. Maybe the data in Yahoo Answers should be preprocessed.

wow,thank you, any reference code for preprocessing?

wow,thank you, any reference code for preprocessing?

https://github.com/SALT-NLP/MixText/blob/master/data/yahoo_answers_csv/Pre-process.ipynb

thank you, i will try. orz


Increment name in prepare_params/1

This may be too simple, but it seems to do what I was expecting:
iex(1)> Tds.Parameter.prepare_params(["test1", "test2"])
[
  %Tds.Parameter{
    direction: :input,
    length: nil,
    name: "@2",
    type: :string,
    value: "test2"
  },
  %Tds.Parameter{
    direction: :input,
    length: nil,
    name: "@1",
    type: :string,
    value: "test1"
  }
]

Plus all the existing tests still pass. :)
Reference to #98

I will check it. Thanks


fix: improve GraphQL tracing compatibility with other tracers

All of the graphql-ruby built-in tracers use the common PlatformTrace module. OpenTelemetry's tracer does not use it directly, but uses the same underlying instance variable and method names.
This can cause conflicts because the new tracing API mixes in trace modules into a single class. This means OpenTelemetry's trace module could override other ones (or vice versa).
This fix avoids conflicts entirely by using the opentelemetry prefix for some methods and instance variables instead of the ones used in PlatformTrace.
An alternative solution would be to refactor this implementation to actually use PlatformTrace. I chose against it some reason originally but I don't remember why 😅 I figured this is just easier for now.
cc @rmosolgo

Going to write a test for this case


Feature  contribution - basic sequence handling in prompt

Feature Idea
I'd like to add simple sequence handling to the prompt parser, to allow for predictable transition through a list of options generation to generation.
At the moment I've got code working locally to support two new prompt keywords:
seq: uses the same separator syntax as random, but progresses through the items in the list in order, looping back to the start when the end is reached.
eg <seq:first thing|second thing|third thing>
wildcardseq: the same idea but applied to the contents of a wildcard file - it will be iterated through in order, with back-to-start looping when the end is reached.
eg <wildcardseq:myWildcardName>
It's not a complex implementation, like the behaviour of combinatorial selection in some other prompt engines, but it does give a basic in-order walk of a list of things, which is what I wanted to use. If this would be more broadly useful, I'm happy to open a PR, just wanted to check in here first as per CONTRIBUTING.md.
Other
No response

Tools -> Grid Generator providers a much nicer interface for this than prompt syntax

Perhaps I missed it in the documentation, but is there a way to get the Grid Generator to loop through entries in a file in order? Or to combine several such loops?


Disable exporting csv when project is running

Temporary fix for #34640
I dived into the backtrace to understand how the crash happened. The data which we get from RemoteDebuggerTCP (which in turn gets from a StreamPeerTCP) sometimes contain more values than usual, from a server or maybe more. During my test it was from physics2d server, maybe because I was in a 2d test scene.
It might be possible to find out how these extra values get added and prevent it. It gets quite complicated at this point.

This is weird, I think we should find a proper fix. The values are not changed in a thread, so getting serialization/deserialization right should be enough.

@Faless
If I'm not mistaken, the part responsible to build the metric to add into the csv starts from this line.
There's the frame.deserialize(p_data); on the third line which is the source of data to build the metric.
I think either the p_data or the deserialization process has problem. Probably it's the deserialization following your comment.
Last time I analyzed it was here that has more data than it should be - a server sometimes contains more items than usual.

I will try looking into the deserialization function.

Superseded by #48917.


Peer testing plans

Peer testing with the other groups is happening next Friday, we need to come up with a plan on how to do it.
My suggestion is to use feat-cstore branch for peer testing as it is extremely simple and works fine right now. It only anonymizes incoming files and writes the anonymized copy to disk, and does not store phi anywhere. But I think even that is sufficient for a demo.
Thoughts?

I agree and I can set up a pipeline to test the cstore branch for Friday. We could even add a simple database interaction to store phi if there is time


Make API consistent

We're using the builder pattern in some cases:

Eip712TransactionRequest
DepositRequest

We need to add consistency across the board. For example, create a WithdrawalRequest in similar fashion.
As part of this taks we should also look into other potential consistency issues.

Hi there! I'm hacking a desktop wallet with Tauri (similar to Electron but in Rust) and I'm trying to create a smart account with a spending limit.
I can't see any reference to AA and the IAccount interface. Am I missing something?
My DMs are open on tg @petra0x if you want to reach out.
Thx!

Hi @0xPetra . We haven't implemented account abstraction yet in the Rust SDK. I think the Python SDK might have it: https://github.com/zksync-sdk/zksync2-python/blob/master/examples/13_deploy_account_create.py

Since I’m using Tauri (which has a Rust backend) I needed this one.
Any timeline or hack I can realize? I’m a noob in Rust. Otherwise I would
try to integrate myself.
On Tue, 11 Jul 2023 at 17:17 Martin Paulucci @.***>
wrote:

Hi @0xPetra https://github.com/0xPetra . We haven't implemented account
abstraction yet in the Rust SDK. I think the Python SDK might have it:
https://github.com/zksync-sdk/zksync2-python/blob/master/examples/13_deploy_account_create.py
—
Reply to this email directly, view it on GitHub
https://github.com/lambdaclass/zksync-web3-rs/issues/83#issuecomment-1631020190,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEZAN22YC2VI5CDQASMXVWDXPVU7BANCNFSM6AAAAAA2EYYMP4
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
https://www.linhub.co/
http://twitter.com/0xPetra
Franco Petra
*0xpetra.com http://0xpetra.com | *.eth
https://etherscan.io/address/0x9fc3b33884e1d056a8ca979833d686abd267f9f8

| *.
https://etherscan.io/address/0x9fc3b33884e1d056a8ca979833d686abd267f9f8lens


Client description toggles between No Name and assigned name over time

When looking at the list of clients, noticed that some clients show the name I have maunally assigned them, then a few minutes late the description field shows No Name.
Screen-shots below

Going to move your images into this one:
https://github.com/cucumber-tony/cucumber-frontend/issues/14
And keep this over there.


Template for chemical and biomolecular characterization data

Currently, the templates are geared towards assays of biological samples. However, there is some interest in submitting characterization data of engineered materials (e.g. polymeric nanoparticles or PNPs). Assays for materials characterization might include NMR spectroscopy, Dynamic Light Scattering, etc.
Currently, it is unclear what attributes are important when submitting data from Dynamic Light Scattering (DLS). For example, if RNA-seq assay template suggests including information like libraryPrep, what is the equivalent for DLS? And what would a generic materials characterization template look like?

This is an interesting problem. Here's a paper I found about CHADA, "a
novel approach for the definition of terminology, classification and
metadata for materials characterization methods":
https://www.sciencedirect.com/science/article/pii/S235249281930087X
Not sure what CHADA stands for.
On Wed, Nov 16, 2022 at 9:10 AM anngvu @.***> wrote:

Currently, the templates are geared towards assays of biological samples.
However, there is some interest in submitting characterization data of
engineered materials (e.g. polymeric nanoparticles or PNPs). Assays for
materials characterization might include NMR spectroscopy, Dynamic Light
Scattering, etc.
Currently, it is unclear what attributes are important when submitting
data from Dynamic Light Scattering (DLS). For example, if RNA-seq assay
template suggests including information like libraryPrep, what is the
equivalent for DLS? And what would a generic materials characterization
template look like?
—
Reply to this email directly, view it on GitHub
https://github.com/nf-osi/nf-metadata-dictionary/issues/213, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AE3WNSF4W3AOJ2JSFBAYOITWIUISHANCNFSM6AAAAAASCO2IDY
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

I think CHADA=CHaracterization DAta
Anyway, after looking through a couple of other databases/sites, I think we can use something that is pretty close to the minimal ISA-TAB-Nano specification.
There is an example for DLS data, which is nice: https://wiki.nci.nih.gov/display/icr/isa-tab-nano#ISATABNano-ISA-TAB-NanoExampleFiles


⚠ Bofrost.es has degraded performance

In 7498a1e, Bofrost.es (https://www.bofrost.es) experienced degraded performance:

HTTP code: 200
Response time: 1514 ms

Resolved: Bofrost.es performance has improved in 823319c after 11 minutes.


Line chart X Axis issue with monthly data

When providing monthly data to the chart without a timestamp, the chart changes the value for the beginning of the month to the end of the previous month.
Expected Behavior
Axis should render on the date specified and should not require a time component.
Current Behavior
Dates provided in the format 2018-01-01 will render as 2017-12-31
Possible Solution
Adding in a timestamp to the date 2018-01-01T12:00:00Z seems to resolve the issue.
Steps to Reproduce (for bugs)
https://codepen.io/damoncool/pen/GdwvQK
Context
Strange unexpected behavior.
Your Environment
https://codepen.io/damoncool/pen/GdwvQK

Hi @DamonCool, thanks for opening this issue!
That's an interesting outcome. We have had some issues before due to the different date formats and how browsers deal with them (using their own new Date() implementation). Actually, you will see how your codepen example doesn't even work in Firefox.
This is a muddy spot where we will need to do some extensive testing to see what is and what is not supported as date formats. What we could do though, is to be super specific when defining the date formats in our docs, so that everybody uses the same formats and we get more consistent and bug-free results.
Thanks a lot for surfacing this issue!

Hi @Golodhros ,
Thanks for the quick response.
I've added a time offset variable to the codepen and you can see that under 8 hours will give unexpected results.  Seems to be a timezone issue.  I'm guessing there is a mixture of GMT(epoch) and local (browser) time in some date math.
If it is a mixture of local time versus GMT in your code, then a specific format would be based on each users timezone, and for their local time.  For companies that have users in multiple timezones, it wouldn't be correct for everyone.  It would probably only be correct for the timezone that the BI team is in.
I'm surprised that this hasn't been noticed before.  Maybe you run your reporting all from GMT?
Not sure about firefox, I don't see any console errors besides a few CORS issues with fonts from coderpen.

Hi @DamonCool, thanks for the follow up!
Actually, we use this charts with different time zones, so I am surprised to see that it is failing under 8 hours.
I still think there is something wrong with that code, as it is not working in Firefox.
I will try to find some time to look at this in more detail, thanks a lot!

Hi @DamonCool,
It seems that yes, if we don't pass a specific timestamp, the date is going to be converted to GMT by the browser.
The solution is to always add a timestamp. I will try to add it to the docs somewhere.
Thanks!


[RFC] POSIX subsys vs libc: playing along or against each other

This is related to https://github.com/zephyrproject-rtos/zephyr/issues/13054 and https://github.com/zephyrproject-rtos/zephyr/issues/12965, and made a separate ticket to capture the essence of the problem and have a focused discussion.
The basic fact:
C library specification is a part of POSIX. See e.g. http://pubs.opengroup.org/onlinepubs/9699919799/functions/memset.html for POSIX specification for memset(). Note that POSIX's libc specification is a superset of ANSI C standard library specification.
In other words, we'll never have a no-nonsense POSIX subsystem, until along with it we provide a decent libc implementation. Currently, Zephyr support 2 libc implementations:

Minimal libc, fully part of Zephyr source tree. But implementing a tiny subset of ANSI C standard library, and far less of POSIX libc.
newlib libc, which is pretty full-fledged implementation of ANSI C and POSIX libc (though some parts are missing, e.g. BSD Sockets API implementation). newlib is an external project, shipping along with toolchain as part of the Zephyr SDK.

The essence of the proposal is: Given that minimal libc is clearly inadequate for POSIX compatibility, make newlib a requirement for POSIX subsystem.
The risk of this solution: This effectively also make Zephyr SDK a requirement for POSIX subsys, i.e. POSIX subsys won't be available for 3rd-party toolchains (unless those toolchains include compatible build of newlib (compatible, because newlib itself comes in different versions, there can be changes/incompatibilities and POSIX subsys may depend on a particular version)).
Alternative solution 1:
Support both newlib and minimal libc. This would require pretty heavy development on minimal libc. And not just development, but development in a way to be compatible with newlib (because otherwise POSIX subsys will be ridden with #ifdef for one vs another libc; instead, we should take newlib as a pattern, and make minimal follow it). Put it in plain words, we either will need to create a header set for minimal which would match that of newlib "from scratch", or just copy it from newlib. And that doesn't even talk about actual code to implement the function. In the end of long (or not so long, if we indeed just copy newlib header set) path, we'll end up with 2 copies of newlib: one in the main tree, another in SDK tree, to maintain. That clearly sounds like monkey work to do once, and then over along.
Alternative solution 2:
Try to preserve minimal support for those parts of POSIX subsys which have it now. And that's literally just pthread and fs. This would require the same nature of work as for Alternative 1 (making minimal libc headers compatible with newlib), but at least it will be bounded.

@galak, @nashif: FYI

Regarding alternatives 1 and 2: there should be clear usecases, and clear stakeholders who will pursue that work.
The usecase of the main proposed solution should be clear: to make POSIX subsys to actually deliver (i.e. be able to build existing POSIX apps against Zephyr), with realistic effort.
And note that selecting the main solution doesn't precluding working on Alternatives 1/2 at later time. Again, the main topic of the proposal is to get really-working POSIX subsys "soon". Once that is achieved, we can "optimize" it for usecases where less than the full newlib is needed (well, they can optimize, the stakeholders who need that).

NOTE: Since we are starting to add the support for more toolchains (e.g. Arm Compiler aka. armclang, ARC MWDT) with their own C standard libraries, we will need consider this not just in the context of newlib but for all supported third-party C standard libraries.

@cfriedt  can you please review and see if this is still relevant

The Newlib LibC and POSIX subsystem were to a large extent playing against each other, unfortunately. The situation up until v3.3.0 was that there were several conflicting definitions between POSIX, the Minimal LibC, Newlib, and PicoLibC. Many of those issues (covered in the LTSv3 POSIX Roadmap) have already been fixed but several still need to be addressed - reverse dependencies, dependency cycles, etc.
Additionally, there is a clear need that the Zephyr POSIX implementation not depend on which C library is being used. There are 3rd-party toolchains and C libraries used that have zero POSIX support, but Zephyr's POSIX subsystem is still expected to work as advertised.
In short, we want to coerce the POSIX subsystem into depending on standard C, eliminate any dependencies in the opposite direction, ensure that the implementation has mostly complete test coverage, and ensure that Zephyr has complete control over POSIX headers, types, supported features, and implementation so that there are no "surprises".
As discussed with the libc maintainers (@stephanosio, @keith-packard), and discussed with the TSC too, the proposed direction is to

adopt PicoLibC as the default C library in Zephyr
bring in more mature POSIX headers from Newlib / PicoLibC (leaning more towards the latter)

those will be copied in-tree into Zephyr
standard POSIX feature macros will be available / can be tied into Kconfig
irrelevant / conditionally compiled header sections will be removed (e.g. remove rtems-specific items)

provide sufficient Zephyr-specific functionality to PicoLibC so that the libc has virtually no reliance on POSIX
disable POSIX definitions, types, etc, in the libc for Zephyr's builds, forcing POSIX control back to the Zephyr Project

We currently do not rely on Newlib for supported POSIX features. However, Zephyr does need to make up a lot of ground in getting up to the embedded and real-time profiles.
Since much of the current direction supersedes this RFC, I'll close but make a mention of it in the LTSv3 POSIX Roadmap.


Role Guard

I would like to know what is the annotation for Roles like we have in nest-keycloak-connect.
Ex:
@ Roles({ roles: ['admin'], mode: RoleMatchingMode.ANY })

Hi @tharbts
We don't currently support Role guards. The only guards we support be found from https://github.com/anonrig/nestjs-keycloak-admin/tree/master/src/guards
If you'd like to implement Role guards, you can always open a pull request!

@tharbts it's been implemented in the latest release.


EVP-2011. Add postgres check to node and js pods

Is this change necessary because the pods don't check for its dependencies before marking them as ready? if the pods fail initially, aren't they marked as ready/live eventually when the dependencies are alive?

Is this change necessary because the pods don't check for its dependencies before marking them as ready? if the pods fail initially, aren't they marked as ready/live eventually when the dependencies are alive?

I think they check it, but doing so means to start the pod and that takes more resources than the init containers. So this should make the startup of TB more quick as there are less pods fighting for resources

Is this change necessary because the pods don't check for its dependencies before marking them as ready? if the pods fail initially, aren't they marked as ready/live eventually when the dependencies are alive?

I think they check it, but doing so means to start the pod and that takes more resources than the init containers. So this should make the startup of TB more quick as there are less pods fighting for resources

But this is due to our testing environment being resource constrained, that shouldn't be a problem in a production setup right? I guess my concern is that we're leaking environment limitations into something that should be environment agnostic and that raises a flag in my head that something is off somewhere.

But this is due to our testing environment being resource constrained, that shouldn't be a problem in a production setup right? I guess my concern is that we're leaking environment limitations into something that should be environment agnostic and that raises a flag in my head that something is off somewhere.

any thoughts on this? could it be that we are fixing the problem in the wrong place? (the answer can be binary, I just don't know! smile )

Sorry for my late reply. I agree that this should not be an issue in production environments meanwhile they have "enough" resources, but right know we don't even have any specification.
Also notice that this is not "happening" in production environment because database is already up and this check does not help for upgrading those components.  Therefore, the main use case of them if at first installation (any environment).
Last comment: I think that adding this type of checks is also environment agnostic and component specific. In other words, this checks tells K8s that this component is not going to be ready because of the component internals, no because of the environment. So, answering to your question: No,  IMO this checks are useful for every environment.
We already have other rules in the helm chart that could be more environment specific like trying to do not run cassandra pods and tb-node pods in the same node of the cluster. But this is another question and better to discuss offline it with a coffee!

Thanks for the answer, now it makes more sense!


loading language parser for treesitter fails because of glibc mismatch in snap version of nvim

Neovim version (nvim -v)
NVIM v0.7.0-dev+890-g574a58220
Vim (not Nvim) behaves the same?
no
Operating system/version
ubuntu 21.10
Terminal name/version
gnome-terminal 3.38.1
$TERM environment variable
xterm-256color
Installation
snap
How to reproduce the issue
snap install --classic nvim
install treesitter plugin (I used spacevim)
echo "[[layers]]
name = 'treesitter'
">> ~/.Spacevim.d/init.toml

And add that to the init.vim
lua << EOF
require'nvim-treesitter.configs'.setup {
  ensure_installed = "maintained",
  sync_install = false,
  ignore_install = { "javascript" },
  indet = { enable = true, },
  highlight = {
    enable = true,
    additional_vim_regex_highlighting = false,
  },
  incremental_selection = {
    enable = true,
    keymaps = {
      init_selection = "gnn",
      node_incremental = "grn",
      scope_incremental = "grc",
      node_decremental = "grm",
    },
  },
}
EOF

Then  launch nvim and let treesitter-nvim install the language plugins.
Expected behavior
No error message on opening a cpp or lua file
Actual behavior
Error detected while processing FileType Autocommands for "*":
E5108: Error executing lua Failed to load parser: uv_dlopen: /snap/core18/current/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64-linux-gnu/libstdc++.so.6)
stack traceback:
        [C]: in function '_ts_add_language'
        ...6/usr/share/nvim/runtime/lua/vim/treesitter/language.lua:33: in function 'require_language'
        .../nvim/1856/usr/share/nvim/runtime/lua/vim/treesitter.lua:38: in function '_create_parser'
        .../nvim/1856/usr/share/nvim/runtime/lua/vim/treesitter.lua:93: in function 'get_parser'
        .../.cache/init.vim/.dein/lua/nvim-treesitter/highlight.lua:107: in function 'attach'
        ...es/.cache/init.vim/.dein/lua/nvim-treesitter/configs.lua:458: in function 'attach_module'
        ...es/.cache/init.vim/.dein/lua/nvim-treesitter/configs.lua:481: in function 'reattach_module'
        [string ":lua"]:1: in main chunk

Could you try whether compiling using musl-gcc solves the problem?
https://www.musl-libc.org/how.html

The following configuration works for me (apt install musl-dev musl-tools)
require'nvim-treesitter.install'.compilers = {"musl-gcc"}
require'nvim-treesitter.parsers'.command_extra_args = {["musl-gcc"] = {"-static"}}

but musl-gcc fails to build C++ and some C headers.

Interesting approach. So we have a plugin scenario here - what about building the treesitter parser  with the same coreXX environment used with nvim?

Would be possible could do that in our CI.

Hm the one I had used core18.
But if I get that right the neovim binary comes with coreXX runtime dependencies - and a lua interpreter with coreXX dependencies - and the lua interpreter triggers the load of the treesitter parsers? I assume they should to avoid hard to track down binary/ABI incompatibilities (even beyond symbol names - i,e. thread local storage structures ...)  use the same or somehow compatible runtime, since they get loaded in the same process.

Maybe we could create a gcc coreXX snap that you could use to compile the parsers that would use the same libc (as a work-around apart from fixes here in the repo)

@theHamsta is right, you need to use coresnap to compile the tree-sitter parsers or install the same version of GLIBC on the host system. In all honesty I am not sure a snap version is worth it these days for applications such as neovim.

Also seems there was changes to how snap references libraries that maybe the culprit

Can you please provide the LD_LIBRARY_PATH for host and the snap nvim using run shell. Also how did install GCC?

No update, no activity.

Facing same issue

FWIW I installed nvim from the ppa instead of snap and this issue went away.

Ubuntu, neovim from snap, treesitter fails on any input event:

Error detected while processing CursorMoved Autocommands for "*":
Error executing lua callback: Failed to load parser: uv_dlopen: /snap/core18/current/l
ib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /lib/x86_64
-linux-gnu/libstdc++.so.6)
stack traceback:
[C]: in function 'ts_add_language'
...1/usr/share/nvim/runtime/lua/vim/treesitter/language.lua:33: in function 'r
equire_language'
.../nvim/2181/usr/share/nvim/runtime/lua/vim/treesitter.lua:39: in function '
create_parser'
.../nvim/2181/usr/share/nvim/runtime/lua/vim/treesitter.lua:94: in function 'g
et_parser'
...start/nvim-treesitter-context/lua/treesitter-context.lua:165: in function '
get_root_node'
...start/nvim-treesitter-context/lua/treesitter-context.lua:402: in function '
get_parent_matches'
...start/nvim-treesitter-context/lua/treesitter-context.lua:728: in function '
fn'

i have same error

i have same error

Hey bro, did you fixed this problem?

It is not an error but rather side effect of the security and build of snaps. For that reason tree-sitter parsers are compiled and bundled with the snap. If you want to control the Gcc version used to build nvim it is best to avoid the snap.


Duplicate entries in the symbol list

If a function or subroutine name is repeated after the end function/subroutine statement, it will create a duplicate entry in the symbol list for this function/subroutine name. This happens because the identifier uses the scope entity.name.function in both cases, which will automatically added to the symbol list by default. But it only makes sense to have a single entry (pointing to the definition line) for the "Goto Symbol" functionality.
Example:
function foo(x,y,z)
  ...
end function foo

If the same highlighting color for both occurences of the identifier should be kept, one of it could be adjusted to use for example an additional meta scope, and the one after end could be excluded from the symbol list via a .tmPreferences file then.
I think I could prepare a PR to fix this (I guess I would add proper meta scopes to the function/subroutine definition line in that case).

@jwortmann Resolved with PR #8 into development. Will keep issue open until these changes are in main.
I've also added entity.name.class to the symbol list in pull request #16. This gives convenient access to class definitions via ctrl+shift+R. We could also consider adding some other entities to the symbol list (program, interface, module, submodule?). Not sure what the user would expect here, but it seems useful if we can easily search for modules via ctrl+shift+R.

@jwortmann Resolved with PR #8 into development. Will keep issue open until these changes are in main.
I've also added entity.name.class to the symbol list in pull request #16. This gives convenient access to class definitions via ctrl+shift+R. We could also consider adding some other entities to the symbol list (program, interface, module, submodule?). Not sure what the user would expect here, but it seems useful if we can easily search for modules via ctrl+shift+R.

To have entity.name.class in the symbol list is definitely useful and your other suggestions look like a good idea to me too 👍

To have entity.name.class in the symbol list is definitely useful and your other suggestions look like a good idea to me too 👍

Type, program, module, and submodule have now been added to the symbol list in development. Left out interface because it would give duplicates with types in the case of user-defined constructors (type :: myType and interface myType).

Type, program, module, and submodule have now been added to the symbol list in development. Left out interface because it would give duplicates with types in the case of user-defined constructors (type :: myType and interface myType).

Resolved as of v1.1.0.

Resolved as of v1.1.0.


insert into olap external table fail

Steps to reproduce the behavior (Required)

create olap table in target cluster
CREATE TABLE starrocks02_target_table (
event_time datetime NOT NULL COMMENT "datetime of event",
event_type int(11) NOT NULL COMMENT "type of event",
user_id int(11) NULL COMMENT "id of user",
channel int(11) NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(event_time, event_type, user_id)
COMMENT "OLAP"
DISTRIBUTED BY HASH(user_id) BUCKETS 10
PROPERTIES (
"replication_num" = "1"
);

create olap external table in source cluster
CREATE EXTERNAL TABLE external_starrocks_table(
event_time DATETIME NOT NULL COMMENT "datetime of event",
event_type INT NOT NULL COMMENT "type of event",
user_id INT COMMENT "id of user",
channel INT COMMENT ""
)
ENGINE=olap
DISTRIBUTED BY HASH(user_id) BUCKETS 10
PROPERTIES
(
"host" = "x.x.x.x",
"port" = "9020",
"user" = "root",
"password" = "",
"database" = "test",
"table" = "starrocks02_target_table"
);

insert into external_starrocks_table
insert into external_starrocks_table values('2021-11-10',27,1001,01),('2021-11-10',27,1001,01),('2021-11-11',31,1002,02);

Expected behavior (Required)
insert into success!
Real behavior (Required)
ERROR 1064 (HY000): data cannot be inserted into table with empty partition.Use SHOW PARTITIONS FROM external_starrocks_table to see the currently partitions of this table.

StarRocks version (Required)
source cluster  2.0.5
target cluster   2.2.2

You should use remote starrocks's rpc_port to create the external table;  it's not convenient currently, we will improve it.


Not compatible with netstandard2.0 (.NETStandard,Version=v2.0).

Not compatible with netstandard2.0 (.NETStandard,Version=v2.0).

I added support for netstandart
you can use https://www.nuget.org/packages/HDWallet.Tron/0.5.0

Thank you~  And by the way, I think it's a very useful suggestion: this project should consider fully supporting xamarin (. Netstandard). Mobile wallet app is very popular

What is missing from full support?

The target version is not required to be the highest

Many projects are incompatible，please target version is  .net standard 2.0

Can't you target 2.1 instead of 2.0


URI schemed database names are interpreted as files

Issue Summary
When passing a database name in the form of a URI, this is interpreted as file, while it could also be an in-memory database. SQLite defines the URI scheme it uses here: https://www.sqlite.org/uri.html
So for example, this will give an error (since the file name :memory: usually can't be created on a filesystem) when trying to create a database:
new sqlite3.Database("file::memory:?cache=shared");

While it's supposed to create an in-memory database with shared cache.
Also if you want to use a named in-memory database (for use with multiple connections), you should be able to do this:
new sqlite3.Database("file:memdb1?mode=memory&cache=shared");

But this actually creates a file on the file system named file. On Windows there's even the particularity that this becomes a file with alternate data stream (since there's a : in the file name) and the data will be stored in the ADS. Very confusing at first.
The line causing this problem is here: https://github.com/TryGhost/node-sqlite3/blob/master/lib/sqlite3.js#L31
Steps to Reproduce
Try: new sqlite3.Database("file::memory:?cache=shared"); this will give an error.
Try: new sqlite3.Database("file:memdb1?mode=memory&cache=shared");.  This will create a file (on Windows) instead of opening the database in-memory.
Version
5.1.2
Node.js Version
v14.18.2
How did you install the library?
Windows, npm install, library is used through TypeORM

More examples may be found here:
SQLite > In-memory Databases And Shared Cache

@bfraterman-tkhsecurity I was facing the same issue but then realized that an URI requires a different mode. Following should work:
new sqlite3.Database("file:memdb1?mode=memory&cache=shared", OPEN_READWRITE | OPEN_CREATE | OPEN_FULLMUTEX | OPEN_URI)


experimental:spacing-between-declarations-with-annotations false positive

Expected Behavior
/**
 * Some comment
 */
@JvmStatic
fun doSomething() {
    // ...
}

For the above code there should be no error reported.
Observed Behavior
Error is reported:
 Lint error > [experimental:spacing-between-declarations-with-annotations] Declarations and declarations with annotations should have an empty space between.

Steps to Reproduce
Described above.
Your Environment

Version of ktlint used: 0.42.1  (The issue was not present in 0.41.0)

Duplicated of https://github.com/pinterest/ktlint/issues/1053

Might be related to #1053

This has been fixed by https://github.com/pinterest/ktlint/pull/1291

This has been fixed by #1291

Closing issue


Hotstar forbidden error

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

[X] I'm reporting a broken site
[X] I've verified that I'm running yt-dlp version 2022.11.11 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region
India
Provide a description that is worded well enough to be understood
Hotstar giving forbidden error though the video are playable in browser
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
yt-dlp -vU https://www.hotstar.com/in/e307-kara
n-wins-the-contract/1000281715
[debug] Command-line config: ['-vU', 'https://www.hotstar.com/in/e307-karan-wins
-the-contract/1000281715']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8 (No VT), erro
r utf-8 (No VT), screen utf-8 (No VT)
[debug] yt-dlp version 2022.11.11 [8b64402] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-2012ServerR2-6.3.9600-SP0
(OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: ffmpeg N-92391-g07bc603757
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.09.24,
 mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Loaded 1723 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releas
es/latest
Latest version: 2022.11.11, Current version: 2022.11.11
yt-dlp is up to date (2022.11.11)
[debug] [hotstar] Extracting URL: https://www.hotstar.com/in/e307-karan-wins-the
-contract/1000281715
[hotstar]<PHONE_NUMBER>: Downloading JSON metadata
[hotstar]<PHONE_NUMBER>: Downloading webpage
[hotstar]<PHONE_NUMBER>: Downloading token
[hotstar]<PHONE_NUMBER>: Downloading JSON metadata
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),
vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, pro
to, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info]<PHONE_NUMBER>: Downloading 1 format(s): sdr-dash-video_h265_fhd_sdr_16717157
54879-hvc1-4-1+sdr-dash-audio_aac_1671711421329-und-mp4a-3-3
[debug] Invoking http downloader on "https://hses6.hotstar.com/videos/plus/ksbkb
tesp/307_f1d13953cf/656/1000281715/v3/1671711421329/b43a7d0f406c8c62118a85d357b5
9afd/seg-video_h265_fhd_sdr_1671715754879-hvc1-4.mp4"
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 3182, in process_info
  File "yt_dlp\YoutubeDL.py", line 2924, in dl
  File "yt_dlp\downloader\common.py", line 446, in download
  File "yt_dlp\downloader\http.py", line 371, in real_download
  File "yt_dlp\downloader\http.py", line 129, in establish_connection
  File "yt_dlp\YoutubeDL.py", line 3692, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

[x]  I've checked that all provided URLs are playable in a browser with the same IP and same login details

Are you sure it plays in browser under the same IP?
❯ yt-dlp https://www.hotstar.com/in/e307-karan-wins%20-the-contract/1000281715
[hotstar] Extracting URL: https://www.hotstar.com/in/e307-karan-wins%20-the-contract/1000281715
[hotstar]<PHONE_NUMBER>: Downloading JSON metadata
[hotstar]<PHONE_NUMBER>: Downloading webpage
[hotstar]<PHONE_NUMBER>: Downloading token
[hotstar]<PHONE_NUMBER>: Downloading JSON metadata
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading MPD manifest
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[hotstar]<PHONE_NUMBER>: Downloading m3u8 information
[info]<PHONE_NUMBER>: Downloading 1 format(s): video_h265_fhd_sdr_1671715754879-hvc1-4+audio_mp4a_65-Unknown-1
[download] Resuming download at byte 4193280
[download] Destination: E307 - Karan Wins the Contract [1000281715].fvideo_h265_fhd_sdr_1671715754879-hvc1-4.mp4
[download]   6.8% of  118.44MiB at    2.06MiB/s ETA 00:53
ERROR: Interrupted by user

yes it is playable in chrome browser on same ip

Exact same issue here. I was just about to open new issue.


dagster pipeline for edx.org program API ingestion

What are the relevant tickets?

https://github.com/mitodl/hq/issues/6191
Description (What does it do?)

This adds edxorg_program_metadata_extract pipeline to ingest the MITx program data from the edx.org API scheduled daily.
It creates two assets: edxorg/processed_data/program_course_metadata and edxorg/processed_data/program_metadata to be ingested by airbyte.
The API credentials are added to vault QA https://vault-qa.odl.mit.edu/ui/vault/secrets/secret-data/show/pipelines/edx/edxorg/edx-oauth-client. Will need to be added to production.
How can this be tested?

export GITHUB_TOKEN=<>
Ensure your github account can access https://vault-qa.odl.mit.edu/v1/secret-data/pipelines/edx/edxorg/edx-oauth-client
Run dagster dev -d src/ol_orchestrate/ -f src/ol_orchestrate/definitions/edx/edxorg_api_data_extract.py
Then manually materialize two assets from dagster UI

The above two assets should be materialized successfully with the following two log info
Extracted 1165 programs....
Extracted 3050 program courses....

@blarghmatey Thanks for the review. Based on your feedback, I've made the changes. Please take a look.
Also, I am still experiencing hvac.exceptions.Forbidden: permission denied, on get https://vault-qa.odl.mit.edu/v1/secret-data/pipelines/edx/edxorg/edx-oauth-client when running this locally.  So I had to pass the secret to OpenEdxApiClient without Vault manually. That worked for me.
export GITHUB_TOKEN=<>
dagster dev -d src/ol_orchestrate/ -f src/ol_orchestrate/definitions/edx/edxorg_api_data_extract.py


add methods to HttpExchange to convert from and to WARC files

add methods to HttpExchange to convert from and to WARC files

cf https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/
and
https://ait.blog.archive.org/post/the-stack-warc-file/


PayPal cash  Generator

Wont let me in the generator to send myself 100 gift card pay palDescribe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Wont let me in the generator to send myself 100 gift card pay pal

@Rebelman24
To be frank, i dont understand how this has anything todo with the addon, so firstly i would request for you expand your description and and provide a bit more context. (How does you not being able to send yourself 100 gift card relate to the addon?)
After you've dont that i'll gladly try to help out.


"View on site" in django admin does not take into account absence of translation

Scenario:

Admin language in EN
Available languages are EN, DE
Post is created with DE translation ONLY

In the scenario above, the "View on site" URL links to a 404 page. The reason for this is that Post's get_absolute_url does not check for current language (which is what the reverse function does).
See below for a monkey-patch fix:
from django.utils.translation import get_language, ugettext, ugettext_lazy as _
from parler.utils.context import switch_language
from djangocms_blog.settings import get_setting
from django.utils import translation

def monkey_get_absolute_url(self, lang=None):
    if not lang or lang not in self.get_available_languages():
        lang = get_language()
    if not lang or lang not in self.get_available_languages():
        lang = self.get_current_language()
    with switch_language(self, lang):
        category = self.categories.first()
        kwargs = {}
        if self.date_published:
            current_date = self.date_published
        else:
            current_date = self.date_created
        urlconf = get_setting('PERMALINK_URLS')[self.app_config.url_patterns]
        if '<year>' in urlconf:
            kwargs['year'] = current_date.year
        if '<month>' in urlconf:
            kwargs['month'] = '%02d' % current_date.month
        if '<day>' in urlconf:
            kwargs['day'] = '%02d' % current_date.day
        if '<slug>' in urlconf:
            kwargs['slug'] = self.safe_translation_getter(
                'slug', language_code=lang, any_language=True
            )  # NOQA
        if '<category>' in urlconf:
            kwargs['category'] = category.safe_translation_getter(
                'slug', language_code=lang, any_language=True)  # NOQA

        request_language = translation.get_language()
        try:
            if lang not in self.get_available_languages():
                # print "Switching language - as it's not valid"
                lang = self.get_available_languages()[0]
                translation.activate(lang)
            link = reverse('%s:post-detail' % self.app_config.namespace, kwargs=kwargs)
        finally:
            # clean up
            translation.activate(request_language)

        return link

from djangocms_blog.models import Post
Post.get_absolute_url = monkey_get_absolute_url```

@petrklus thanks for spotting it! would be able to contribute the fix as PR?
In case, please open against release/0.9 branch
Thanks!

@petrklus Good patch, but there is an error when it fails in scenario when <category> is enabled as posts show default. It is because you have set check for <slug> before check for <category> but <category> also contains <slug> word in the urlconf and therefore it is matching to <slug> even if <category> set which is causing errors.


Tests for DefaultConnectionService

What does this PR do?

Adds tests for the DefaultConnectionService class.

Description of tasks to be completed.

[x] Use robolectric to test the DefaultConnectionService

How to test this PR?

Run the test files that were created.

Any addition comments.
N/A

Why does the following test (circled) take 7 seconds to run? Is this an indication of a problem?

Why does the following test (circled) take 7 seconds to run? Is this an indication of a problem?

@shahzaibj Seems the test that's run first always takes longer. I just changed the name so that the one you marked was run last.


How can I get the scrape document?

After I scrape a document with node-metainspector and analyze it, I'd like to access the received HTML. How can I do this?

Code below should do it
var Metainspector = require('node-metainspector');

var client = new Metainspector('https://www.google.com/');
client.on('fetch', function() {
  // Do whatever you need with the document here now that we have fetched it
  console.log(client.document);
});

client.fetch();

And is it plaintext or is it an object?
Can I do client.document.getElementById()? because that would make my life lovley.

Its a string but you can use client.parsedDocument which is the body parsed by cheerio which allows you to find elements jquery style. To find a specific element with an id you would use client.parsedDocument("#element")

Alright thanks a lot!


Think about to fix the issues found by adapter checker

I am an automatic service that looks for possible errors in ioBroker and creates an issue for it. The link below leads directly to the test:
https://adapter-check.iobroker.in/?q=https://raw.githubusercontent.com/Black-Thunder/ioBroker.melcloud

[x] [E605] No actual year found in copyright. Please add "Copyright (c) 2021 black-thunder<EMAIL_ADDRESS>at the end of README.md
[x] [E701] No actual year found in LICENSE. Please add "Copyright (c) 2021 black-thunder<EMAIL_ADDRESS>at the start of LICENSE

Thanks,
your automatic adapter checker.
P.S.: There is a community in Github, which supports the maintenance and further development of adapters. There you will find many experienced developers who are always ready to assist anyone. New developers are always welcome there. For more informations visit: https://github.com/iobroker-community-adapters/info

Thanks, that all bugs have been fixed.

Thanks, that all bugs have been fixed.


Make token sets not editable when unchecked

Is your feature request related to a problem? Please describe.
I would love if an unchecked tokenset was uneditable. I have multiple tiers and jump between files to work on separate libraries, but sometimes I have selected the wrong tokenset and change or add what I need before I realize that I'm in the wrong tokenset.. Can anyone relate?
Describe the solution you'd like
Add a setting to disable editing for unchecked token sets
From Slack conversation: https://figmatokens.slack.com/archives/C02JAL48UDN/p1654681040084709

@mikekamminga is this still an active request of yours or can we archive? I think the need you had here is being addressed differently.


Tray icon disappears

Short description
Multiple tray icons disappear after left clicking the TaskbarX tray icon. This problem persists after restarting TaskbarX.
Steps to reproduce
Steps to reproduce the bug:

Start up the computer
TaskbarX starts automatically
Left click the TaskbarX tray icon

Steps to restore the taskbar and system tray to its original state (without TaskbarX):

Stop TaskbarX
Restart Windows Explorer

Screenshots and Detailed Description
Usually the tray icon looks like this:

Complete screen
The system tray seems to be broken once I left click the TaskbarX tray icon. It disappears together with all other icons after the TaskbarX tray icon. Only one empty icon remains. In the screenshot, you cannot see the mouse pointer but I'm pointing at this empty tray icon. It doesn't have an icon (small image), it doesn't have a tooltip and it doesn't do anything when it is clicked:

Complete screen
There are no icons after that one empty icon. When I reopen the tray, it becomes smaller:

Complete screen
If you stop TaskbarX, the empty tray icon will disappear, but the other tray icons will still be missing. To fix this problem, you have to restart Windows Explorer. All missing tray icons will reappear.
But as soon as you start TaskbarX again, the same icons as before will disappear once more and the empty icon will be there again, too.
It is possible to fix the empty tray icon and display the TaskbarX tray icon correctly again. But this won't fix the other missing tray icons. To do that, you have to disable the TaskbarX tray icon in the Configurator, click Apply, enable the tray icon and click Apply once more. Now the empty tray icon is gone and the TaskbarX tray icon is displayed and working, but the other tray icons that disappeared before are still missing.
Desktop:

OS: Windows 10 Home
Version: 21H1
Build: 19043.1052
TaskbarX: <IP_ADDRESS>

Yeah, I have this same issue after recently purchasing and using TaskbarX myself. Once TaskbarX starts, some of my icons disappear entirely, one goes blank with no icon but still has its place, and others move from their place on the visible task tray into the little "more icons" pop out area instead; for example, the icons from HWiNFO64 for CPU and GPU temperature move from their position.
OS: Windows 10 Pro (Version	10.0.19043 Build 19043)
TaskBarX: <IP_ADDRESS>
Purchased and installed via Windows Store.

To be able to focus on major/current bugs I had to close all issues. The issues section has gotten very messy to a point that I couldn't follow it anymore. Most issues are duplicates, very old or are already fixed/solved. Feel free to reopen this issue if you still have this issue on the latest version of TaskbarX to get my attention. All feature requests have been put in a list here: 💡| Feature requests #870
I hope you understand ❤


Update README.md

Codecov Report

Merging #4 (b4b0cd0) into main (853b349) will not change coverage.
Report is 1 commits behind head on main.
The diff coverage is n/a.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@           Coverage Diff           @@
##             main       #4   +/-   ##
=======================================
  Coverage   60.88%   60.88%
=======================================
  Files           4        4
  Lines         271      271
=======================================
  Hits          165      165
  Misses         80       80
  Partials       26       26

Flag
Coverage Δ

unittests
60.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


SeleniumHelpers broken for Odyssey

In the course of debugging #180, ran into the below error. It appears that in the course of abstracting SeleniumHelpers into a base class, we lost the ability to supply a web driver to Odyssey page objects (e.g. PortalPage, Search etc all require a driver on instances to function properly).
In order to avoid breaking high-level SeleniumHelpers, which appears to now be used in IA and WI scrapers, we should subclass SeleniumHelpers in odyssey namespace and update all references in Odyssey page classes to use the downstream version:
# court_scraper/platforms/odyssey/selenium_helpers.py

from court_scraper.base.selenium_helpers import SeleniumHelpers as BaseHelpers

class SeleniumHelpers(BaseHelpers):
    def __init__(self, driver):
        self.driver = driver

To reproduce the bug:
court-scraper search --with-browser -p wa_adams -c 08-2-00866-9

Executing search for wa_adams
Traceback (most recent call last):
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/bin/court-scraper", line 8, in <module>
    sys.exit(cli())
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/court_scraper/cli.py", line 74, in search
    results = runner.search(**kwargs)
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/court_scraper/platforms/odyssey/runner.py", line 45, in search
    data = site.search(case_numbers=case_numbers)
  File "/Users/tumgoren/.local/share/virtualenvs/court-scraper-etl-B5T25nbt/lib/python3.7/site-packages/court_scraper/platforms/odyssey/site.py", line 55, in search
    portal_page = PortalPage(self.driver)
TypeError: PortalPage() takes no arguments

Has there been any progress on this?


BED-4391: Make v5.12.0 migration idempotent

Description
Fixes the part of our v5.12.0 migration that is not idempotent
Motivation and Context
Migrations should be written as idempotent whenever possible to ensure they can be reapplied safely
How Has This Been Tested?
Deleted the migration record for v5.12.0 and restarted dev after running into a migration issue. This solves it.
Types of changes

Chore (a change that does not modify the application functionality)
Bug fix (non-breaking change which fixes an issue)
Database Migrations

Checklist:

[x] I have met the contributing prerequisites

Assigned myself to this PR
Added the appropriate labels
Associated an issue: https://github.com/SpecterOps/BloodHound/issues/672
Read the Contributing guide: https://github.com/SpecterOps/BloodHound/wiki/Contributing

[x] I have ensured that related documentation is up-to-date

Open API docs
Code comments (GoDocs / JSDocs)

[x] I have followed proper test practices

Added/updated tests to cover my changes
All new and existing tests passed

We should also make the insert idempotent as well correct?

It seems pretty safe as-is, do you have a suggestion on making it more correct?

It seems pretty safe as-is, do you have a suggestion on making it more correct?

It'll conflict on second attempt to insert because of the singleton
Adding ON CONFLICT DO NOTHING ought to fix that.

It seems pretty safe as-is, do you have a suggestion on making it more correct?

It'll conflict on second attempt to insert because of the singleton Adding ON CONFLICT DO NOTHING ought to fix that.

Didn't run into it on local, but seems like a straightforward improvement, I'll get that added