Datasets

This is a very rough work-in-progress for adding Dataset handling. This should probably go before #121, as there are some fairly fundamental and disruptive changes proposed here. This is just a sketch so we can discuss the ideas more concretely, so I'd really appreciate any feedback people might have here. Here's major changes:

We change the server CLI to take a single positional argument which is the config file. Within the config file we have a list DATASETS, which contains a list of directories. Each directory is assumed to correspond to exactly one dataset, and has a prescribed format. For now, we just have a 'variants' sub directory, but we will also have a 'reads' directory too.
Within the variants sub-directory, each directory corresponds to a VariantSet. For now, we inspect the directory to decide whether we should allocate a WormtableVariantSet or a TabixVariantSet. I've outlined what I think is a better idea in the comments based on 'Shards' of the variant set.
I've introduced the concept of a DataObject which should be the superclass of all objects that represent an aggregation of data in some way. So, Dataset, VariantSet, ReadSet, ReferenceSet, all of these should be DataObjects. Hopefully we can push a lot of the tedious housekeeping and file system walking code up here. Hopefully this will help make testing a bit easier also.  I think we should explicitly model this as a hierarchy, as this might make the tricky iteration/page token problems a bit simpler, as we can then view it as a tree-traversal problem and maybe generalise the code.

The definition is in reads.avdl, and there are back references from a bunch of other objects. However, the inconsistency of dataset vs callSet, variantSet, referenceSet, readGroupSet` is irritating...

Ooops, just clobbered my own changes with a git push -f from a stale branch. Will reinstate them later  when I get out my laptop.

Hmm, I was under the impression that each data type had it's own
*Set object at the root. and that they are not shared and maybe
fundamentally different.
I think open source development works better when code is being
written rather than discussed on conference calls..
Jerome Kelleher<EMAIL_ADDRESS>writes:

The definition is in reads.avdl, and there are back references from a bunch of
other objects. However, the inconsistency of dataset vs callSet, variantSet,
referenceSet, readGroupSet` is irritating...
—
Reply to this email directly or view it on GitHub.*

My understanding (and I could be completely wrong) is that the dataset is a top level container for read and variant data. So we'd have
dataset1/
       + readGroups
             + readGroupSet1
             + readGroupSet2
             + ....
       + variantSets
             + variantSet1
             + ....
dataset2/
       + ....

The dataset container is then used to determine access privileges --- is user X entitled to read from dataset Y?

Jerome Kelleher<EMAIL_ADDRESS>writes:

My understanding (and I could be completely wrong) is that the dataset is a top
level container for read and variant data. So we'd have

and so little is actually written down.  I guess the reference
manual is the minutes from the conference calls.
This seems very problematic.  What if you have multiple variant
sets produce independently?

The dataset container is then used to determine access privileges --- is user X
entitled to read from dataset Y?

This is very wrong ;-)  Access needs to be determined by
functions applied at multiple levels, not by a fixed container.
For instances, in TCGA germ-line variants are protected, somatic
variants are not.  Then they decided that whole genome somatic
show be considered protected, but not exome ...
sigh ....

Well, it looks like this is what we have for now. I'd vote on implementing it (even if it's just a straw man) and use this to figure out exactly what is wrong with the model. At least then we have something concrete.
Would someone else like to weigh in here? @adamnovak you've done some work with datasets, what's your understanding of how they work/what they do?

YES.  implementing is the best way to validate!!
I will dig more and start a ticket.
Jerome Kelleher<EMAIL_ADDRESS>writes:

Well, it looks like this is what we have for now. I'd vote on implementing it
(even if it's just a straw man) and use this to figure out exactly what is
wrong with the model. At least then we have something concrete.
Would someone else like to weigh in here? @adamnovak you've done some work with
datasets, what's your understanding of how they work/what they do?
—
Reply to this email directly or view it on GitHub.*

Dataset has not been formally defined. Google said it is used for permission control and added to the GA4GH APIs. Practically, Google uses a dataset for a project (e.g. 1000g, PGP, SGDP, etc). I have always wanted to define dataset in a more precise way.

This is very wrong ;-)  Access needs to be determined by functions applied at multiple levels, not by a fixed container. For instances, in TCGA germ-line variants are protected, somatic variants are not. Then they decided that whole genome somatic show be considered protected, but not exome ...

Under this dataset model, TCGA could be divided into multiple datasets: "TCGA germline", "TCGA somatic", etc.

The definition is in reads.avdl, and there are back references from a bunch of other objects. However, the inconsistency of dataset vs callSet, variantSet, referenceSet, readGroupSet, etc. is irritating...

A few people at Broad have concerns over the naming callSet. We rarely call a sample as a callSet. I personally do not like readGroupSet.

I think open source development works better when code is being written rather than discussed on conference calls..

Couldn't agree more.

YES.  implementing is the best way to validate!!

I also vote to implement dataset.The way @jeromekelleher has described is proper as I understand.

I think on line 81 in variants.py, it probably should be variantSetId instead of ariantSetId.
Regarding dataset, I would go as far as the following post:
https://github.com/ga4gh/schemas/pull/74
Regarding enabling a more modularized and parallel approach, the following would should provide that flexibility:
https://github.com/ga4gh/schemas/issues/143
Basically a dataset and/or project can be formed ad hoc, while others can be become gold standards.  The above I think should provide that flexibility and granularity.
Paul

Heng Li<EMAIL_ADDRESS>writes:

Dataset has not been formally defined.

nor has id ;-)

Google said it is used for permission control and added to the
GA4GH APIs. Practically, Google uses a dataset for a project
(e.g. 1000g, PGP, SGDP, etc).

This is a simplification beyond the reality of the ugliness
of the data and the projects.

I have always wanted to define dataset in a more precise way.

I am old fashioned, I want to define what things before you
specify the data structures.
It's seems problematic to have back pointers to `dataset' in
contained objects.  It reduce flexibility.  These things are
better answerable via queries, not stored in object.

Under this dataset model, TCGA could be divided into multiple datasets: "TCGA
germline", "TCGA somatic", etc.

This is a very serious flaw.  When policy changes, the entire
data collection has to be reorganized.  Access control needs to
be a function that takes user and the desired objects and
returns yes or no.  This leaves actually defining access control
policy outside of GA4GH.

YES. implementing is the best way to validate!!

I also vote to implement dataset.The way @jeromekelleher has described is
proper as I understand.

and time for a PR.

Thanks very much for the input on this @diekhans and @lh3. I appreciate that the dataset abstraction has many problems, especially with respect to access control. However, we need to implement some sort of access control,  datasetIds are already in the schema, and assigning access permissions to users based on datasetIds is simple and at least minimally useful. I hope we can implement this simple model first, and discuss how we can improve on it as we gain experience with its deficiencies.

As long as the access control isn't defined in the dataset
record, only the dataset id is a parameter to it.
But at the moment, I would be happy to have zero access control  and
more API implemented ;-)
Jerome Kelleher<EMAIL_ADDRESS>writes:

Thanks very much for the input on this @diekhans and @lh3. I appreciate that
the dataset abstraction has many problems, especially with respect to access
control. However, we need to implement some sort of access control, datasetIds
are already in the schema, and assigning access permissions to users based on
datasetIds is simple and at least minimally useful. I hope we can implement
this simple model first, and discuss how we can improve on it as we gain
experience with its deficiencies.
—
Reply to this email directly or view it on GitHub.*

There are folks working on implementing OAuth in the reference server @diekhans --- hopefully, we can have access control and more API!

Jerome Kelleher<EMAIL_ADDRESS>writes:

There are folks working on implementing OAuth in the reference server @diekhans
--- hopefully, we can have access control and more API!

Excellent!
My two-cents on how authorization should be implemented is by
functions that take a user and an object and return yes/no.
While something at the level of data set might work for many
cases, one can imagine all kinds of interesting filter.  For
instance, one could only allow access to certain regions of the
genome.   Given a man/woman a data structure, and they will ask for
enhancement, give the a function and they can build all kinds of
cool things.

Agreed @diekhans, this is a great way to do it, but it needs fleshing out at the schema/security policy level. Let's keep it simple in terms of the implementation for now.
Anyone have any feedback on my code here? @pashields, @Naburimannu and @dcolligan this is a sort of shell of the structure I was talking about the other day. Any thoughts on how it's looking so far?

Jerome Kelleher<EMAIL_ADDRESS>writes:

Agreed @diekhans, this is a great way to do it, but it needs fleshing out at
the schema/security policy level. Let's keep it simple in terms of the
implementation for now.

My argument would be that access control function vs declarative
data structures are far simpler (although I wouldn't go near as
far as call them on anything lower than at *Set type of object
with many clean use cases).

Anyone have any feedback on my code here? @pashields, @Naburimannu and
@dcolligan this is a sort of shell of the structure I was talking about the
other day. Any thoughts on how it's looking so far?

As others have pointed out, enforcing access control at the dataset level would be simple, but coarse.  (Which is certainly a good characteristic of a POC implementation.)
Is it true that no object within a dataset can be shared across dataset?  That is, could a, say, variantSet be in two of the same datasets?  If so, then this model will duplicate potentially a lot of data.  (An alternative would be a model where every object had a top-level directory, and then instances named by id within it, i.e.:)
variantSets/
    variantSetId1/
    variantSetId2/
callSets/
...

But you guys have a better handle on the data model so take my comments with a salt shaker.
If we do end up going with this model, be sure to add what datasets we are using to our brand new html page!

Dataset is very ill-defined at this point.  We can help shape it!
Danny Colligan<EMAIL_ADDRESS>writes:

As others have pointed out, enforcing access control at the dataset level would
be simple, but coarse. (Which is certainly a good characteristic of a POC
implementation.)
Is it true that no object within a dataset can be shared across dataset? That
is, could a, say, variantSet be in two of the same datasets? If so, then this
model will duplicate potentially a lot of data. (An alternative would be a
model where every object had a top-level directory, and then instances named by
id within it, i.e.:)
variantSets/
variantSetId1/
variantSetId2/
callSets/
...
But you guys have a better handle on the data model so take my comments with a
salt shaker.
If we do end up going with this model, be sure to add what datasets we are
using to our brand new html page!
—
Reply to this email directly or view it on GitHub.*

Closing this as #158 needs to be resolved first.


USB connectivity problems on macOS

Setup: MacBook Pro running Mojave connected to Joulescope hardware through the provided A-B USB cable.  Joulescope UI 0.4.6.
If I plug in JS hardware after starting the app, no readings come through. Quitting the app and restarting fixes the issue. Not a big deal.
My MBP sleeps after a period of keyboard inactivity. When I awaken it, the app no longer updates readings. A restart of the App fixes the problem.
Thanks for the report, John!

The reported issue is fixed in the 0.5.0 software.  However, we now have a related (but much less frequent) issue over on pyjoulescope.


⚠ Bofrost.fr has degraded performance

In d7f64bb, Bofrost.fr (https://www.bofrost.fr) experienced degraded performance:

HTTP code: 200
Response time: 994 ms

Resolved: Bofrost.fr performance has improved in 4e073ae.


[Feature] Context instead of msg.sender

Feature Description

Convert the usage of msg.sender  to OZ Context's sender() to support gasless operations using a paymaster.
Additional Information

💯 percent correct, _msgSender(), my bad in writing the ticket without checking 😁


Failed to change kernel

Type: Bug
Can't run my notebook powershell code because it won't let me change kernels.   Defaults back to SQL displaying the error message below.
Failed to change kernel. Kernel SQL will be used. Error was: Notebook process exited prematurely with error code: 1. StdErr Output: Traceback (most recent call last): File "c:\Program Files\Azure Data Studio\resources\app\extensions\notebook\resources\pythonScripts\startNotebook.py", line 1, in  import notebook.notebookapp ModuleNotFoundError: No module named 'notebook.notebookapp'
Googled found several articles that suggested to reinstall python and jupyter which i tried and did not fix issue
Azure Data Studio version: azuredatastudio 1.45.0 (8eff468100f02157f2219335f7a566d14451ca49, 2023-07-24T18:22:12.106Z)
OS version: Windows_NT x64 10.0.19044
Restricted Mode: No
Preview Features: Enabled
Modes:

System Info

Item
Value

CPUs
11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz (8 x 1805)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
undefined

Memory (System)
15.73GB (5.70GB free)

Process Argv

Screen Reader
no

VM
0%

Extensions (11)

Extension
Author (truncated)
Version

admin-pack
Mic
0.0.2

admin-tool-ext-win
Mic
0.1.3

agent
Mic
0.49.0

cms
Mic
0.9.3

dacpac
Mic
1.14.0

datavirtualization
Mic
1.13.2

import
Mic
1.5.5

machine-learning
Mic
0.12.0

powershell
ms-
2023.6.0

profiler
Mic
0.14.0

whoisactive
Mic
0.1.5

Seems duplicate of https://github.com/microsoft/azuredatastudio/issues/23954
@RobbinsbkC can you try recommendation from that issue and use same thread for follow up?
I will close this as duplicate.

That recommendation didn't help.   Thanks for closing my case without any solution thought.

@RobbinsbkC the other issue will be used for investigation, so please upvote and post a comment there if none of the recommendations helped your case. Thanks!


Bug located in TcpOslServer.start() method

🔍 Before submitting the issue

[X] I have searched among the existing issues
[X] I am using a Python virtual environment

🐞 Description of the bug

when calling start on already evaluated project, execution is stuck because of exec_started_listener waiting for server notification PROCESSING_STARTED that is not delivered

Solution:

check project status before sending start command (similarly as we do with stop command)
check run_scenario or another property?

already_finished = False
project_info = self._get_basic_project_info()

run_scenario = project_info["projects"][0]["run_scenario"]

if run_scenario == "FINISHED":
   self._logger.warning("`START` command wasn't sent, project was already evaluated.
                          If you wish to reevaluate it anyway, call `RESET` at first.")
   return

📝 Steps to reproduce

load already evaluated project
run start
(runs fine if reset is called before start)

from ansys.optislang.core import Optislang, examples

parametric_project = examples.get_files("calculator_with_params")[1][0]
osl = Optislang(
    project_path=parametric_project,
    loglevel="INFO",
)
# osl.reset()
osl.start()
osl.dispose()

💻 Which operating system are you using?
Windows
📀 Which ANSYS version are you using?
2023R2dev
🐍 Which Python version are you using?
3.7
📦 Installed packages
n/a

@PanekOndrej I'd expect that some other notification or error is returned. Something like "NOTHING_PROCESSED". Isn't that the case?

@PanekOndrej I'd expect that some other notification or error is returned. Something like "NOTHING_PROCESSED". Isn't that the case?

You're right, server notification NOTHING_PROCESSED is returned indeed. We only need to add this to the list the ExecStarted listener's registered notifications and callback then.

@PanekOndrej Alright, sounds good.


Add support for virtual networks

Add missing vnet resource
Related issues: #1602 cloudflare/cloudflare-go#913

as this is a new resource, can you please swap over to using the autogenerated docs by following the example in #1677? it should only require:

adding the descriptions to the schema fields
adding either example usage
adding the import example
running make docs

I'll review the functionality in the AM.

as this is a new resource, can you please swap over to using the autogenerated docs by following the example in #1677? it should only require:

adding the descriptions to the schema fields
adding either example usage
adding the import example
running make docs

I'll review the functionality in the AM.

@jacobbednarz created examples and generated the docs

looks like the update comment test is failing
TF_ACC=1 go test $(go list ./...) -v -run "^TestAccCloudflareTunnelVirtualNetwork_" -count 1 -parallel 1 -timeout 120m -parallel 1
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
=== RUN   TestAccCloudflareTunnelVirtualNetwork_Exists
--- PASS: TestAccCloudflareTunnelVirtualNetwork_Exists (14.70s)
=== RUN   TestAccCloudflareTunnelVirtualNetwork_UpdateComment
    resource_cloudflare_tunnel_virtual_network_test.go:118: Step 2/2 error: Error running apply: exit status 1

        Error: error updating Tunnel Virtual Network "4760e5b2-c737-4b8d-98f9-9ce33b581d7c": error unmarshalling the JSON response error body: unexpected end of JSON input

          with cloudflare_tunnel_virtual_network.xuysfsklky,
          on terraform_plugin_test.tf line 2, in resource "cloudflare_tunnel_virtual_network" "xuysfsklky":
           2: resource "cloudflare_tunnel_virtual_network" "xuysfsklky" {

--- FAIL: TestAccCloudflareTunnelVirtualNetwork_UpdateComment (18.02s)
FAIL
FAIL	github.com/cloudflare/terraform-provider-cloudflare/internal/provider	33.292s
?   	github.com/cloudflare/terraform-provider-cloudflare/tools/cmd/changelog-check	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/tools/cmd/maintainer-only-file-check	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/tools/cmd/tf-log-check	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/version	[no test files]
FAIL
make: *** [testacc] Error 1

from the HTTP traffic, it looks like the virtual network ID is missing in the PATCH request.

couple of small things and we can get this one merged in

All green now

acceptance tests all passing here
TF_ACC=1 go test $(go list ./...) -v -run "^TestAccCloudflareTunnelVirtualNetwork_" -count 1 -parallel 1 -timeout 120m -parallel 1
?   	github.com/cloudflare/terraform-provider-cloudflare	[no test files]
=== RUN   TestAccCloudflareTunnelVirtualNetwork_Exists
--- PASS: TestAccCloudflareTunnelVirtualNetwork_Exists (13.33s)
=== RUN   TestAccCloudflareTunnelVirtualNetwork_UpdateComment
--- PASS: TestAccCloudflareTunnelVirtualNetwork_UpdateComment (24.03s)
PASS
ok  	github.com/cloudflare/terraform-provider-cloudflare/internal/provider	37.941s
?   	github.com/cloudflare/terraform-provider-cloudflare/tools/cmd/changelog-check	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/tools/cmd/maintainer-only-file-check	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/tools/cmd/tf-log-check	[no test files]
?   	github.com/cloudflare/terraform-provider-cloudflare/version	[no test files]

thanks for getting this over the line!


Allow export-analyzer to ingest multiple files

TL;DR
Currently, export-analyzer can analyze one file at a time.  When having to analyze all exports of a key server, it would save a ton of time to call the tool once as opposed to once per zip.
Design
Proposal
Create a new flag like "--files" for a comma separated list, or "--zips-location" to point to a directory wherein the zips reside.  Then the tool would ingest all files and output all at once.

@bschlaman - if there is a specific use case that we can help solve as well, let us know


Configure in Coinbase/yml

es6-modules/config.yml in coinbase:master

I can't see any purpose for this change.


Upgrade Arrow dependency to 14.0.0

Arrow 14.0.0 is out and with support for REE, which can be used to export Velox Constants through the ABI.

LGTM, we should move this to resolve dependency as well at some point to allow for use of system/conda arrow. (or is this something for after the fbthrift changes?)

I was wondering is this file was there for a reason, but it seems like it's just legacy? Let me take a stab at moving it around.


[DataCap Application]

Version
1
DataCap Applicant
Dot Well
Project ID
1
Data Owner Name
Dot Well
Data Owner Country/Region
China
Data Owner Industry
Information, Media & Telecommunications
Website
http://www.dotwell.com/
Social Media Handle
https://mp.weixin.qq.com/s/uiYTh4Y8GlVIrXHAI0PW8w
Social Media Type
Slack
What is your role related to the dataset
Dataset Owner
Total amount of DataCap being requested
5PiB
Expected size of single dataset (one copy)
100TiB
Number of replicas to store
5
Weekly allocation of DataCap requested
800TiB
On-chain address for first allocation
f3wk5smnr2msm4dwd7upkfgt25uwe6umwbht4kild4r4v5nco3borspqkuu3tvvkwperl4ztnvgupsu7vavhiq
Data Type of Application
Public, Open Commercial/Enterprise
Custom multisig

[ ] Use Custom Multisig

Identifier
No response
Share a brief history of your project and organization
Shenzhen DotWell Culture Communication Co., Ltd. (DotWell) is a diversified and integrated creative agency.
Services provided by Dotwell:
Live-action shooting/CG animation/KV/brand strategy/new media integrated marketing/MCN/game art & R&D/self-media business cooperation/column packaging/pan-entertainment content creation/H5/scene marketing exhibition display, etc.

Is this project associated with other projects/ecosystem stakeholders?
No
If answered yes, what are the other projects/ecosystem stakeholders
No response
Describe the data being stored onto Filecoin
Live shooting video/game video/brand video/product video/corporate video/entertainment video/plane/CG animation/KV/game art & R&D

Where was the data currently stored in this dataset sourced from
My Own Storage Infra
If you answered "Other" in the previous question, enter the details here
No response
If you are a data preparer.  Please include tooling used and technical details?
No response
If you are not preparing the data, who will prepare the data?  (Provide name and business)
No response
Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.
https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2240
It has been approved before.

Please share a sample of the data
(http://www.dotwell.com/work/#section2) you can watch the videos from us in the link.

Confirm that this is a public dataset that can be retrieved by anyone on the Network

[X] I confirm

If you chose not to confirm, what was the reason
No response
What is the expected retrieval frequency for this data
Yearly
For how long do you plan to keep this dataset stored on Filecoin
1 to 1.5 years
In which geographies do you plan on making storage deals
Greater China
How will you be distributing your data to storage providers
HTTP or FTP server, Shipping hard drives
How did you find your storage providers
Slack, Partners
If you answered "Others" in the previous question, what is the tool or platform you used
No response
Please list the provider IDs and location of the storage providers you will be working with.
HK        f01730296
Guangdong f02825338   f02639675
SIchuan   f0109903
Sichuan   f02201915

How do you plan to make deals to your storage providers
No response
If you answered "Others/custom tool" in the previous question, enter the details here
No response
Can you confirm that you will follow the Fil+ guideline
Yes

The information that you offered has been checked. It looks great. Some other questions for you:
What's the relationship between you and the organization?
Could you send an email to<EMAIL_ADDRESS>with your official domain in order to confirm your identity? And show the screenshot here.
Has this data been stored before?If so, why are you choosing to store it again?

-I am the manger of the data apartment in our company.
-Pls check the eamil.
-We applied the application before, https://github.com/filecoin-project/filecoin-plus-large-datasets/issues/2240
It was signed by the ff government, but it is no DC left because the allocators. So we apply again and hope get your support.Z

checker:manualTrigger

checker:manualTrigger

We will add more SPs in the following step. Hope to get  the next approval. Thanks.

checker:manualTrigger

checker:manualTrigger

checker:manualTrigger

@Cloudyia  Can you share the information about your SP? Something wrong with the CID report.

f03074592

f03074583

We find new SP partners last round. Here are the information. Hope we can get approved for the next round.

checker:manualTrigger

checker:manualTrigger

Because no report show for us. Here are the SP deal sistuation.If you have any question please let me konw. Hope we can get approved for the next round. Thank you so much.

Because there is no CID Checker Report,I can't check the whole sp information. So i can't grant 2PiB to you as the rule. The max DC is no more than 1PiB.

       Because no report show for us. Here are the SP deal sistuation.If you have any question please let me konw. Hope we can get approved for the next round. Thank you so much.

@Cloudyia  Can you explain the ''No active deals found for this client'' problem? Hope you can fix it.

@Cloudyia  Due to NO CID Checker Report in these rounds. I will not approve the application until it is fixed.

checker:manualTrigger

checker:manualTrigger

checker:manualTrigger

checker:manualTrigger

checker:manualTrigger

checker:manualTrigger

No CID checker report . Closed

checker:manualTrigger

checker:manualTrigger


[REQ] Default skins always as the first option in categories

I noticed that the default CS2 skins are not in first place in the categories.

Wouldn't it be possible for the default CS2 skin to always be in first position? This applies to all categories, skins, knives, gloves and agents.

Update v1.4.0.


CHAOS REPORT: Gmail is getting submitted without adding the '@' symbol or a proper address

Is there an actual issue here?

[] I have verified this isn't part of ChaosWeb's intentional disorder

Describe the unexpected!

When I tried to submit a Gmail without adding a proper address, it still got submitted! If you assign me the task, I will do my best to improve this.
Confirm

[X] I understand ChaosWeb is intentionally designed to be messy
[X] I agree to follow this project's Code of Conduct
[X] I found a genuine bug amidst the chaos
[X] I'm enjoying the mess!
[X] I'm GSSOC'24 Extd registered Contributor
[X] I'm Hacktoberfest registered Contributor

Its chaotic, if you will carefullt notice then all the fields are incorrect so that was done purposely
Sorry for this time!
You can search for some other issue which I will be happy to assign


Fix #25 by providing an Akka Stream Source graph stage for Kinesis.

This graph stage provides an akka source graph based on the actor model provided by the reactive kinesis consumer.
If a consumer is configured, an akka stream Source can be obtained via a simple Kinesis.source("consumer-name").
Every message that flows through the stream needs to be committed explicitly.
An integration test is added that relies on localstack running on the same node (I did not add the localstack startup logic as part of this PR).
Steps to run the test:

Set this env var, since the localstack does not support CBOR: export AWS_CBOR_DISABLE=true
Start localstack (I use the docker-compose with: docker-compose -f localstack.yml up) in default configuration (Kinesis on port 4568, Dynamo on port 4569)
sbt it:test

It was not as easy to come up with a good integration test, since a Kinesis Source usually never finishes. To have a finite test, I use Flow.take which ends the stream after a defined amount of entries. The interplay of different readers/shard consumers could be different between runs, but the test expectation should be met.

Coverage decreased (-6.008%) to 80.769% when pulling f3577970db64dc586e0cc443d2ef9435102f0f95 on aquamatthias:mv/akka_stream into 5bff161c2b419a2e1baca1ccdbfdd8adae2b1e6e on WW-Digital:master.

Thanks @aquamatthias! Regarding the localstack stuff - I'm in the process of adding that and hooking it up to travis in https://github.com/WW-Digital/reactive-kinesis/pull/34
I've got a lot of travelling to do this week but I'll dig into this properly over the next couple of weeks 👌

This is really cool!

Ok it's been a month - so sorry @aquamatthias - now things have settled down it's high up on my list!!

I've finished the integration tests finally - so they run in travis now against localstack.
This is next...

Created a branch off this to rebase master and refactor the integration tests. My intention is to find a middle ground between what I did and what you did. There are pros and cons of each!
https://github.com/WW-Digital/reactive-kinesis/tree/feature/aquamatthias/akka-streams
Right now it's building and running in travis - but not quite finished yet.
Not spent much time looking at the main code yet / reviewing...
Seen this fail a couple of times in the travis builds:
[info] - should maintain the read position in the stream correctly *** FAILED ***
[info]   Vector("1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30", "31", "32", "33", "34", "35", "36", "37", "38", "39", "40", "41", "42", "43", "44", "45", "46", "47", "48", "49", "50", "51", "52", "53", "54", "55", "56", "57", "58", "59", "60", "61", "62", "63", "64", "65", "66", "67", "68", "69", "70", "71", "72", "73", "74", "75", "76", "77", "78", "79", "80", "81", "82", "83", "84", "85", "86", "87", "88", "89", "90", "91", "92", "93", "94", "95", "96", "97", "98", "99") had size 99 instead of expected size 100 (KinesisSourceSpec.scala:135)

May be my refactoring - I'll keep prodding :)

@markglh Sounds good. Let me know if I there is something I can help with.
The failed test sounds like a shaky test - I will try to investigate.

Thanks @aquamatthias - you should be able to merge my branch into yours without conflicts 👌


Centralize the definition of isMemberwiseInitialized()

This utility was defined in Sema, used in Sema and Index, declared in
two headers, and semi- copy-pasted into SILGen. Pull it into
VarDecl::isMemberwiseInitialized() and use it consistently.

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge

@swift-ci please smoke test and merge


Remix Vite Plugin triggers watchers during Build

Reproduction
I'm using the Vite Plugin for building an app (and deploy it to Production), and I have two issues:

vite build seems to trigger watchers on all files in my workspace.
Using pnpm on Gitlab, I configured viteConfig.server.watch.ignored to include **/.pnpm-store/**.
This prevents the CI to crash because of too much inotify watchers.
(See kinda related issue https://gitlab.com/gitlab-org/gitlab/-/issues/434329)

I'm not sure if the watchers are due to the Remix Plugin or to Vite itself, but I have to put context here.
I don't understand why vite build would need to watch anything on the workspace?

Until version 2.4.0 it was possible to configure which files had to be ignored, but since 2.4.0 (in this commit), the configuration of the Server does not overwrite User config but does an entire replace instead.

Configuration
export default defineConfig({
  server: { port: 3000, {
    watch: {
     ignored: ["**/.pnpm-store/**"] // This is not read by the plugin anymore
    }
  },
  plugins: [
    tsConfigPaths(),
    remix({
      ignoredRouteFiles: ["**/.*", "**/*.css", "**/*.test.{js,jsx,ts,tsx}"],
    }),
  ],
});

System Info
(System is not relevant here, the issue is on Gitlab CI.)

  System:
    OS: macOS 14.2
    CPU: (10) arm64 Apple M1 Max
    Memory: 16.54 GB / 64.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 21.3.0 - ~/.nvm/versions/node/v21.3.0/bin/node
    Yarn: 1.22.21 - ~/.nvm/versions/node/v21.3.0/bin/yarn
    npm: 10.2.4 - ~/.nvm/versions/node/v21.3.0/bin/npm
    pnpm: 8.7.6 - ~/Library/pnpm/pnpm
  Browsers:
    Chrome: 120.0.6099.109
    Safari: 17.2
  npmPackages:
    @remix-run/dev: 2.4.0 => 2.4.0
    @remix-run/eslint-config: 2.4.0 => 2.4.0
    @remix-run/express: 2.4.0 => 2.4.0
    @remix-run/node: 2.4.0 => 2.4.0
    @remix-run/react: 2.4.0 => 2.4.0
    @remix-run/server-runtime: 2.4.0 => 2.4.0
    vite: 5.0.10 => 5.0.10

Used Package Manager
pnpm
Expected Behavior

Remix Vite plugin should not trigger watchers during build.

It should be possible to configure which files should be ignored using viteConfig.server.watch.ignored.

Actual Behavior

Remix Vite Plugin triggers watchers for all files in workspace (except node_modules/ and .git/ which are Vite defaults)

viteConfig.server.watch.ignored is entirely skipped by the Remix Vite Plugin.

I just took a quick look at how SvelteKit managed the build, and it seems that they don't use vite.createServer but just vite.build:
https://github.com/sveltejs/kit/blob/c0f3c403b774df72b113646559e915e4f53a8566/packages/kit/src/exports/vite/index.js#L740

Quick and dirty fix (until implementation using vite.build) could be to set viteConfig.server.watch to null.

Interesting, thanks for reporting the issue.
The code you're referencing (around viteChildCompiler = await vite.createServer(...)) is currently necessary for both dev and build and it's used for transforming code independently from Vite (but still benefiting from Vite-powered transform).
Note that Remix makes use of vite.build API for main build already, but viteChildCompiler is still necessary and I think this is somewhat internal detail (and thus not inheriting all user config via { server: { ... viteUserConfig.server } } feels still right to me):
https://github.com/remix-run/remix/blob/9edcf88d3c25cdabbe628782927cae2c735242c6/packages/remix-dev/vite/build.ts#L71
To fix this, viteConfig.server.watch: null makes sense and I think that should be safe at least for build case. Thanks for the suggestion!

Fixed by #8320.


Feature request: Make it free to rename things in the anvil

I like to name my toolboxes and things like that so I can tell them apart, but it requires XP to do so. Given that the player isn't getting a lot of XP through the normal gameplay I think it would be nice if you could rename things in the anvil for free.
I know FTB Genesis has something that lets you do that, if you need to find a source.
Thank you!

Oh yes, this is a great idea, I also love to do that, will add right away

Still didn't figure out a way to make this work

 mini_me0000 on Discord says

well you can get tons of XP crushing  ores...

 yarden.zamir on Discord says

True but there is no good reason to require that

 yarden.zamir on Discord says

In general I prefer the player won't have xp in the first place

 yarden.zamir on Discord says

As that's a very organic creatures thing in my mind

 yarden.zamir on Discord says

pending/contributors/code

 coolandonrs on Discord says

oh this is just a mixin I bet

 coolandonrs on Discord says

Give me a moment

 coolandonrs on Discord says

not even a huge mixin at that

 coolandonrs on Discord says

oh its not even a mixin you can just change a private static final var and its done

 coolandonrs on Discord says

yarden.zamir is it worth me making a pr for this that uses some nice simple reflection shenanigans to get it done?

 yarden.zamir on Discord says

yea i think so

 yarden.zamir on Discord says

make sure to wrap it in a feature

 coolandonrs on Discord says

wdym

 yarden.zamir on Discord says

like this

 yarden.zamir on Discord says

if (feature('Make quartz grow with dripping')) {

 coolandonrs on Discord says

what?

 coolandonrs on Discord says

you know I'm doing this in java just to double check yes

 yarden.zamir on Discord says

ah alright, missed that

 yarden.zamir on Discord says

isn

 yarden.zamir on Discord says

't that sort of thing possible with kjs reflection?

 coolandonrs on Discord says

kjs reflection is java reflection

 coolandonrs on Discord says

I just hate js

 yarden.zamir on Discord says

very limited though.

 yarden.zamir on Discord says

fair

 coolandonrs on Discord says

alright: https://github.com/jasperalani/ptdye-plus/pull/5

 yarden.zamir on Discord says

Awesome. kikis2111 for you to merge when you have time

 coolandonrs on Discord says

I haven't tested it. Originally that was because I forgot I could, but my gradle runClient is erroring terribly and I don't have the time to set up a pseudo-prod atm so make sure you do that. Primary error condition is if there is a SecurityManager set up. Just means mixins instead of reflection which isn't bad at all in theory.

 coolandonrs on Discord says

It's also just as easy to change any of these cost factors or the maximum name length if either of those are desired.

 kikis2111 on Discord says

coolandonrs i have replied to the pr's

 coolandonrs on Discord says

Replied to both; I’ll do the comments for this one once I get home.

 coolandonrs on Discord says

kikis2111 sorry, forgot about this again but I'm committing documentation momentariliy (once my computer cooperates) just so you know. I saw your comments on Find a replacement for quark crate and replace it's PR and I'll try to figure that out shortly.

 yarden.zamir on Discord says

i think at some point the only job the anvil will have is to rename things, as thats the only thing there is no alternative for

 coolandonrs on Discord says

So it turns out that some I would need to commit several henious crimes to get reflection to work in the correct manner. I was looking into directly calling inbuilt JVM methods to give you an idea of how terrible it can get.

Fortunatley mixin can do it for us fairly easily so once I get that opening I think I'll open a new PR and close the other one.

 coolandonrs on Discord says

Actually kikis2111 would you rather me open a new PR for the new approach and close the other one or just do rebase shenanigans to use the old one?

 coolandonrs on Discord says

Oh well this is curious

 coolandonrs on Discord says

This is actually practically impossible

 coolandonrs on Discord says

The anvil won't recognize that it has a recipe if the cost is 0 it would seem

 coolandonrs on Discord says

So you have to rework anvil internals. (which is possible, when I say practically impossible I am exaggeratting)

 coolandonrs on Discord says

Oh wait it might be easier then thought it might just be a single if statement that needs to be bypassed

 coolandonrs on Discord says

It also turns out that these are just entirely ignored as far as I can tell, funnily enough. Just got a working solution, going to submit it as a new PR and close the old one kikis2111.

 coolandonrs on Discord says

Well, it "works". Discovered a few bugs I'll iron out first

(still will take a level if you have one, but doesn't require it, and the  UI still shows that it costs 1 level, though it will show it green even if you have no xp)

 yarden.zamir on Discord says

can we just hide the number in that case

 yarden.zamir on Discord says

with fancymenu if needed even?

 engineersmith_4628 on Discord says

is the issue with it being private? Because you can probably use an accesstransformer to make it public... possibly - I know you can use access transformers on classes as that is what I had to do for my current task with animation

 coolandonrs on Discord says

So I typed this whole thing out before remembering that I discovered that COST_RENAME doesn't even seem to be referenced, so you have to do mixins anyway. I left the rest of the message as a spoiler incase you are curious.

||In a way? The issue (why reflection won't work) is that java.lang.reflect.Field#modifiers is private. It seems, even, that the java.lang.reflect namespace has a special exception when it comes to reflection and getDeclaredFields() will not return private fields for that class.
So, while it's entirely possible with access wideners, what we almost certainly end up widening is one of the following depending on how cooperative the JVM wishes to be with our crimes  (from least cooperative to most):

java.lang.Class#getDeclaredFields0(boolean)* and one of:
java.lang.reflect.Method#methodAccessor
java.lang.reflect.Method#getMethodAccessor()
java.lang.reflect.Method#aquireMethodAccessor()*
java.lang.reflect.Field#modifiers
net.minecraft.world.inventory.AnvilMenu#COST_RENAME||

 coolandonrs on Discord says

We plausibly won't need to once the issue is acutally fixed

 coolandonrs on Discord says

I discovered it after this, but the issue is actually much more generic. You may use an anvil with any amount of xp, and it will always succeed. It will then take as much xp from you as you have, up to the actual cost.

 coolandonrs on Discord says

which is significantly less desired, and I'm not entirely sure why it's happening.

 engineersmith_4628 on Discord says

So; you just have to give the player a level just before it takes it

 coolandonrs on Discord says

Oh that's a little gross

 coolandonrs on Discord says

I don't like that one

 coolandonrs on Discord says

Plus it wouldn't work

 coolandonrs on Discord says

The AnvilMenu will only allow you to withdraw the item if you already have the XP, thought its only taken when you actually withdraw the item. See AnvilMenu#mayPickup vs AnvilMenu#onTake

 engineersmith_4628 on Discord says

could we just give the player max xp; and then hide their xp level <:betsy:1175697772880609320> then you wouldn't need any java

 coolandonrs on Discord says

https://cdn.discordapp.com/emojis/996357458169692230.webp?size=48&name=despair&quality=lossless

 coolandonrs on Discord says

This does have absurd potential to be abused with the disenchantment drain or whatever its called

 engineersmith_4628 on Discord says

and I would abuse it so much

 yarden.zamir on Discord says

Can we disable the drain interaction with players?

 yarden.zamir on Discord says

If we make player xp vistually none exsistant that would be perfect

 coolandonrs on Discord says

That is a much worse solution then just making the anvil cost nothing; but probably

 yarden.zamir on Discord says

Id like that anyway

 yarden.zamir on Discord says

As i don't want the player to be able to store xp

 yarden.zamir on Discord says

on themselves

 coolandonrs on Discord says

At that point just remove XP and Anvils as a whole and use kubejs/java to make a lookalike that is just a barebones clone with only renaming functionality

 coolandonrs on Discord says

lookalike part optional

 yarden.zamir on Discord says

That sounds much harder

 coolandonrs on Discord says

Much easier, actually

 engineersmith_4628 on Discord says

Why does anything need to be renamed anyway. To abuse the quest system by renaming trades and then crafting them into others?

 yarden.zamir on Discord says

Go ahead then, if you make sure it doesnt mess up the things we do want then sure

 coolandonrs on Discord says

if (StringUtils.isBlank(this.itemName)) {
if (itemstack.hasCustomHoverName()) {
k = 1;
i += k;
itemstack1.resetHoverName();
}
} else if (!this.itemName.equals(itemstack.getHoverName().getString())) {
k = 1;
i += k;
itemstack1.setHoverName(Component.literal(this.itemName));
}```
basically just reimpl this

 yarden.zamir on Discord says

it checked names that were none italic iirc

 yarden.zamir on Discord says

before you moved it it id based

 yarden.zamir on Discord says

Renaming is cool

 yarden.zamir on Discord says

I want renaming to be a thing

 yarden.zamir on Discord says

I dont mind if we have a different way to rename

 coolandonrs on Discord says

Actually the easiest way would be a command

 coolandonrs on Discord says

I don't know how agreeble that is, but a block shouldn't be too much harder

 engineersmith_4628 on Discord says

Cries in block entities

 yarden.zamir on Discord says

Lets reset for a second, where are we at now? We made it free but not visually free?

 coolandonrs on Discord says

We made paying optional for everything

 coolandonrs on Discord says

you pay as much as you have

 yarden.zamir on Discord says

right, optional not free

 coolandonrs on Discord says

for literally any anvil operation

 yarden.zamir on Discord says

That part is merged or not yet?

 coolandonrs on Discord says

Its in a draft PR

 coolandonrs on Discord says

Wait the hell, anvils don't have a block entity

 coolandonrs on Discord says

fascinating

 engineersmith_4628 on Discord says

Ooo question

 engineersmith_4628 on Discord says

is it possible to have a custom name that isn't italic?

 yarden.zamir on Discord says

Not in survival i think

 coolandonrs on Discord says

Yeah  if you touch the NBT directly

 yarden.zamir on Discord says

Though i wish it was possible

 engineersmith_4628 on Discord says

ooo thanks

 coolandonrs on Discord says

Actually

 engineersmith_4628 on Discord says

I got an item that I have data I want in it - but using custom name causes it to go italic - it isn't the worse but it doesn't look the part

 coolandonrs on Discord says

The easiest way to do this (which would be a little jank) is to use a command /rename or something with /tellraw formatting

 coolandonrs on Discord says

You can also edit lore in such a manner just as easily

 yarden.zamir on Discord says

In code? The text components can be easily modified for that yes

 engineersmith_4628 on Discord says

I tried setting italics to false; but nothin;

 engineersmith_4628 on Discord says

Animation Library

 yarden.zamir on Discord says

i think it might be italic

 coolandonrs on Discord says

OH wait thats just wow.
If your chill with a command yarden.zamir then it would take like 3 seconds to implement

 coolandonrs on Discord says

(assuming it works first try)

 coolandonrs on Discord says

(which it wont)

 yarden.zamir on Discord says

What do you mean by command?

 yarden.zamir on Discord says

As a way to rename?

 engineersmith_4628 on Discord says

I wouldn't think it would be that hard since there is setCustomName or something

 coolandonrs on Discord says

yes

 yarden.zamir on Discord says

Not really, command is not too player facing

 coolandonrs on Discord says

Well yes but you could do it with tellraw so the full power of TextComponent is behind you

 coolandonrs on Discord says

Just make a kubejs block that runs a command on the item inside

 coolandonrs on Discord says

https://cdn.discordapp.com/emojis/1059585348675575858.webp?size=48&name=Clueless&quality=lossless

 yarden.zamir on Discord says

Item.of('minecraft:chest', 1, '{BlockEntityTag:{LootTable:"ptd:drg_cave"}}').withName(Text.gold("14.5% of expedition loot").italic(false)),

 coolandonrs on Discord says

anyway I need to go to bed, I'll try to make an anvil clone when I get some free time.

 coolandonrs on Discord says

So I've basically gotten it working. Fully working anvil clone that works with two minor issues: It's untextured (fine) and it doesn't actually do the renaming. It lets you rename an item, but the item doesn't actually get renamed

 coolandonrs on Discord says

Should be an easy fix (it probably won't be)

 coolandonrs on Discord says

Alright it actually was fairly easy, just one minor bug left (Emptying the text doesn't put it back to its default name, though if you put in just a space it will)

 coolandonrs on Discord says

actually this might be vanilla behavior weirdly enough

 coolandonrs on Discord says

Alright, assuming thats vanillla behavior it now works (though it says that it costs a level and you don't have it) So I'm committing and changing to a normal PR (and will search for a way to remove that UI bit)

 coolandonrs on Discord says

yarden.zamir I did make it so the anvil clone can't break, by the way (though its changeable via a const bool). Is this desired or no?

 yarden.zamir on Discord says

yes sounds good. the new one is just for renaming, need to think of a theme for it as it shouldn't actually be an anvil

 coolandonrs on Discord says

as of the latest commit it is just a textureless block that looks (shape and UI) and sounds like an anvil, and is called an anvil.

 yarden.zamir on Discord says

yea, talking about the next steps then. making a block that renamed things is the hard part. next is removing fixing behavior and then name texture and model

 coolandonrs on Discord says

Fixing behaviour is gone already

 coolandonrs on Discord says

But don't forget about the screen

 yarden.zamir on Discord says

removing the second slot and changing it to take title from a separate Lang file

 yarden.zamir on Discord says

or is there more that needs to be done there?

 coolandonrs on Discord says

Yeah so basically you have to remake it from scratch to do that. Title is done, though, thats different.

As well as removing the second slot requiring a bunch of reimpl.

 yarden.zamir on Discord says

hm. still practical to do though?

 coolandonrs on Discord says

Yeah, just much more work. What I did was easier because I just copied the anvil and made changes. If we want to change at that left it becomes less "change the anvil" and more "start from scratch".

 coolandonrs on Discord says

I have discovered how to remove the level text though. I just need to coerce it into functioning correctly.

 coolandonrs on Discord says

I need to take a break, don't know if I'll work more on it today, but if someone else wants to attempt to remove the level text you want to mixin to net.minecraft.client.gui.screens.inventory.AnvilScreen#renderLabels. You probably want to make your mixin within a abstract class that extends net.minecraft.client.gui.screens.inventory.ItemCombinerScreen{T} so you can get access to this.menu.

 kikis2111 on Discord says

Hey coolandonrs where are we on this?

 kikis2111 on Discord says

in your last comment on github you said we need someone to make a texture for the new anvil but if its the same as regular anvil can we not copy and use that texture

 coolandonrs on Discord says

Ah, I totally forgot about it again. This is what I get for working on several projects more or less simultaneously.

Yeah, more or less unless we go down the dedicating renaming block route that yarden.zamir was saying he’d prefer.

 kikis2111 on Discord says

oh i see

 kikis2111 on Discord says

no problem

 kikis2111 on Discord says

ok so lets get some artists on this

 kikis2111 on Discord says

pending/contributors/art contributors/art hey guys, would anyone be able to make a texture for a 'Renaming Block'
Basically it acts like an anvil but its a whole new block

 autistic.sack.of.friends on Discord says

i'll try whipping something up in BB

 autistic.sack.of.friends on Discord says

hmm

 autistic.sack.of.friends on Discord says

 kikis2111 on Discord says

sure

 kikis2111 on Discord says

like a workbench'

 autistic.sack.of.friends on Discord says

a WORKBENCH?

 autistic.sack.of.friends on Discord says

we can't be having that

 kikis2111 on Discord says

lmao

 autistic.sack.of.friends on Discord says

i think ideally renaming would be something you can do anywhere, but a station is fine

 kikis2111 on Discord says

yeah going to need a block either way

 coolandonrs on Discord says

I suggested a command but yarden said he’d greatly  prefer a block

 autistic.sack.of.friends on Discord says

wouldn't an inventory tool be ok

 autistic.sack.of.friends on Discord says

you can just shove it in storage if you want

 autistic.sack.of.friends on Discord says

i mean if we implemented the "tools are free, in that they don't take up any inventory space if you get rid of them and can be summoned anywhere" idea

 kikis2111 on Discord says

yeah we never really finalized that idea

 kikis2111 on Discord says

i think were going to need a block and tool though

 kikis2111 on Discord says

for immersion

 kikis2111 on Discord says

and automation

 autistic.sack.of.friends on Discord says

how would you automate renaming in a way that isn't just how create enchantment industry does it

 autistic.sack.of.friends on Discord says

the only thing i can think of is letting you specify a pattern

 autistic.sack.of.friends on Discord says

like renaming item A to item B's id

 autistic.sack.of.friends on Discord says

or renaming items to a continously incrementing counter

 autistic.sack.of.friends on Discord says

would it be possible for it to be a rollover station, like create's fluid drain

 autistic.sack.of.friends on Discord says

or is that going too far

 kikis2111 on Discord says

yeah idk maybe a trade that requires items to be numbered or something, could be a interesting challenge

 autistic.sack.of.friends on Discord says

not if we make a machine that does it

 autistic.sack.of.friends on Discord says

maybe something like "comparator input signal"

 kikis2111 on Discord says

yeah

 autistic.sack.of.friends on Discord says

like you can specify a pattern when renaming an item

 autistic.sack.of.friends on Discord says

and there's a lot of different replace codes

 kikis2111 on Discord says

yeah

 kikis2111 on Discord says

lets do the block first

 autistic.sack.of.friends on Discord says

trade where you need to crunch da numbers for an external company

 kikis2111 on Discord says

then this

 autistic.sack.of.friends on Discord says

yeah i'm on it

 autistic.sack.of.friends on Discord says

 kikis2111 on Discord says

some kind of redstone timer

 autistic.sack.of.friends on Discord says

maybe a little shelf inside there

 autistic.sack.of.friends on Discord says

holding writing supplies and such

 kikis2111 on Discord says

little pot for live squids that can be milked

 kikis2111 on Discord says

tiny live ink squids

 autistic.sack.of.friends on Discord says

of COURSE

 autistic.sack.of.friends on Discord says

how else would you make it

 autistic.sack.of.friends on Discord says

worlds most smallest squid

 autistic.sack.of.friends on Discord says

dooms you to internal transparency 1000 years

 autistic.sack.of.friends on Discord says

not even blockbench can render this correctly...

 autistic.sack.of.friends on Discord says

 autistic.sack.of.friends on Discord says

the squid no longer has any water

 autistic.sack.of.friends on Discord says

a few vials of ink back there

 autistic.sack.of.friends on Discord says

this desk is now certified "renaming"

 autistic.sack.of.friends on Discord says

 autistic.sack.of.friends on Discord says

still needs some work but there's a channel which items would roll along to get renamed in theory, if we went with that angle

 autistic.sack.of.friends on Discord says

kikis2111 so, what do you think of this model?

 yarden.zamir on Discord says

I like that, maybe a button in inventory then?

 yarden.zamir on Discord says

we can have items for each letter and then you combine them in create automated crafting in a row in a dynamic recipes to generate words and sentances

 yarden.zamir on Discord says

that would be epic and also useless because its easier to use a nametag

 yarden.zamir on Discord says

if we can find a way to make this useful it could be fun

 yarden.zamir on Discord says

but likley initial renaming is just inherantly a manual task

 yarden.zamir on Discord says

which means it can be a block or an inventory button function thing

 yarden.zamir on Discord says

Im thinking another inventory tab

 autistic.sack.of.friends on Discord says

im still hoping for a very silly math puzzle

 autistic.sack.of.friends on Discord says

or maybe automated account processing

 kikis2111 on Discord says

Lmao

 autistic.sack.of.friends on Discord says

like you read off a book and complete a trade with mobs renamed to different things

 kikis2111 on Discord says

Looks great asof

 autistic.sack.of.friends on Discord says

like the book is full of

pig Terrance
cow Betsy

 autistic.sack.of.friends on Discord says

and you need to provide named mobs as the book dictates

 autistic.sack.of.friends on Discord says

I am really curious what kind of infrastructure that would create

 autistic.sack.of.friends on Discord says

you'd need at minimum to trigger one of four deployers from an item passing through four filters

 autistic.sack.of.friends on Discord says

and then cycle the item, in order to go gather the next mob

 autistic.sack.of.friends on Discord says

along with gather the other half of the book line and use it to create a name tag

 autistic.sack.of.friends on Discord says

this is all a really silly puzzle

 autistic.sack.of.friends on Discord says

what's even the point of automated naming of mobs outside of making a pile of mobs that never despawn

 kikis2111 on Discord says

asof can you finalize a block texture for now, and also an icon for an inventory tab and ill make that happen

 autistic.sack.of.friends on Discord says

im thinking a pencil will be fine

 kikis2111 on Discord says

ya

 autistic.sack.of.friends on Discord says

pencils is hard

 kikis2111 on Discord says

hm

 autistic.sack.of.friends on Discord says

we could go for something like this

 kikis2111 on Discord says

can you show it to size

 autistic.sack.of.friends on Discord says

i don't have the asset for the tabs

 autistic.sack.of.friends on Discord says

could you screenshot the inventory window with the tabs real quick

 kikis2111 on Discord says

well just zoom out a bit so its smaller

 kikis2111 on Discord says

if thats possible

 autistic.sack.of.friends on Discord says

 autistic.sack.of.friends on Discord says

the=re

 autistic.sack.of.friends on Discord says

smaller

 kikis2111 on Discord says

 kikis2111 on Discord says

thx, maybe make the eraser a bit darker?

 autistic.sack.of.friends on Discord says

tried the cursor quick, doesn't look good in the context of the personal assembler

 kikis2111 on Discord says

hm yeah

 kikis2111 on Discord says

maybe a stylized i

 autistic.sack.of.friends on Discord says

 kikis2111 on Discord says

like at the start of a book

 kikis2111 on Discord says

https://i.stack.imgur.com/rcTox.jpg

 autistic.sack.of.friends on Discord says

the shading isn't good here, i'll try fixing that first then move on to nice stylized designs

 autistic.sack.of.friends on Discord says

 autistic.sack.of.friends on Discord says

that's looking ok...

 kikis2111 on Discord says

yeah

 kikis2111 on Discord says

i agree it clashes with the screwdriver

 kikis2111 on Discord says

maybe it doesnt

 autistic.sack.of.friends on Discord says

something abstract just looks out of place

 autistic.sack.of.friends on Discord says

everything else is a thing

 autistic.sack.of.friends on Discord says

yeah

 kikis2111 on Discord says

pencil is fine for now for me

 autistic.sack.of.friends on Discord says

 autistic.sack.of.friends on Discord says

more shiny, not sure if that helps

 kikis2111 on Discord says

can you add the tab to this image

 kikis2111 on Discord says

and possibly make an enabled version


Revert: [SPARK-47040][CONNECT] Allow Spark Connect Server Script to wait

What changes were proposed in this pull request?

This PR reverts

https://github.com/apache/spark/commit/73da134849b7d2bbd170bb1530dcdf6d5f94f0da
https://github.com/apache/spark/commit/a6bed5e9bcc54dac51421263d5ef73c0b6e0b12c

Why are the changes needed?

See https://github.com/apache/spark/pull/45090#issuecomment-1953579857
I would recommend removing --wait, just use env var and update https://spark.apache.org/developer-tools.html to mention that.
There are several server components in Spark, like Thrift Server, History Server, Master, Worker, only adding --wait on the Connect Server makes things inconsistent, while env vars approach is adopted widely in Spark for different development/test purposes. Additionally, --wait is not quite accurate, it also affects stdout/stderr redirection not just process exiting.
Does this PR introduce any user-facing change?

No, it's an unreleased feature
How was this patch tested?

Review
Was this patch authored or co-authored using generative AI tooling?

No

... because it's not properly documented ...

you can search SPARK_PREPEND_CLASSES on https://spark.apache.org/developer-tools.html

... just use env var and update https://spark.apache.org/developer-tools.html to mention that.

And I do emphasize in the PR description to update the docs to mention the env var SPARK_NO_DAEMONIZE usage.

Additionally, SPARK_NO_DAEMONIZE is listed in spark-env.sh.template
➜  conf cat spark-env.sh.template | grep SPARK_NO_DAEMONIZE
# - SPARK_NO_DAEMONIZE  Run the proposed command in the foreground. It will not output a PID file.

Thank you for pinging me @dongjoon-hyun.
Just my humble opinion. Although users can use this feature, it is primarily intended for use by the Spark Connect developers, right? It's low-maintenance, so I think we can take care of their habits. Both sys environments and command line options have their advantages. If a developer frequently needs to switch between the frontend and backend during the development process, suffixed --wait would be more convenient as it can be easily backspaced or auto-suggested.
Well, unfortunately, --wait is '$1' now. How about making it more lenient about the position?

I don't fully understand why this is an issue. It's a convenience option for users. Folks are used to run --help on command line scripts and see useful output. For developers in particular looking at the spark log while running the Spark Connect service is very helpful. Knowing that an environment variable exists is much more effort than calling --help.
The part that makes it even harder for me is that this has caused no issue, no harm, nobody was complaining about it from the user base, but we debate wether or not environment variables are better than command line options.

Hi, @grundprinzip . Yes, we are debating in a healthy way. There is nothing wrong here. This is a part of healthy community process .
For this part, I would not exclude @pan3793 from the user base.

nobody was complaining about it from the user base,

I absolutely appreciate the feedback. I spend most of my time developing Spark Connect and I can only attest that this option is very useful to me. I have not seen any reasoning besides "I prefer environment variables" and "the other servers don't have this option".
I invite everyone to spend time developing features for Spark Connect (in particular Scala) and it will be obvious why this option is really helpful. Maybe the lack of experience in this dev-loop is driving this decision.
But I stand by my opinion that improving developer efficiency and first experience should not be underestimated.

"the other servers don't have this option"

@grundprinzip That's the point, it violates the consistency. The option is exposed to end users, from the user's point of view, if ./sbin/start-connect-server.sh --wait works, why ./sbin/start-thriftserver.sh --wait does not work? What if I want to add a --debug option to ./sbin/start-thriftserver.sh --debug, things go to mess without restrictions.

@pan3793 @dongjoon-hyun @grundprinzip How about creating an umbrella jira to support option --wait in the start scripts under sbin/ dir?

There are just a few of them.
Having such a command line option would be helpful. Many users just run ./sbin/sript -h to see the available options and they may never be aware of the env variables.
These new tasks would be great for new Spark developers.

There are several server components in Spark, like Thrift Server, History Server, Master, Worker, only adding --wait on the Connect Server makes things inconsistent, while env vars approach is adopted widely in Spark for different development/test purposes.

Mind pointing out the examples explicitly?

Mind pointing out the examples explicitly?

@HyukjinKwon To clarify, the assumption is limited to dev/test purposes, not to end users. Examples:

SPARK_PREPEND_CLASSES
SPARK_GENERATE_GOLDEN_FILES
SKIP_MIMA
SKIP_RDOC
SKIP_PYTHONDOC
SKIP_SQLDOC

@gengliangwang I think it's also good if the community reaches a consensus to promote the --wait to all services, like SHS, STS.
Another simple way is mentioning SPARK_NO_DAEMONIZE in sbin/ scripts' help message.

@grundprinzip I appreciate your great contributions to Spark Connect and improving development efficiency, I just started exploring this new feature. I strongly agree that we should pay attention to improving development efficiency. But let's evaluate it relatively carefully for user-facing changes. Spark is a great product that has been brilliant for over a decade, we should respect the existing services, tools, and user habits when making user-facing changes, and I believe a consistent user experience is an important element for such a great product.

The examples you mentioned are mostly for development purpose. The examples I provided are more for user-facing. I think this --wait is the similar case. I think it's fine to add an easier option for those options users are expected to use often.

@HyukjinKwon so you are good to promote the --wait to all services, right?

@HyukjinKwon so you are good to promote the --wait to all services, right?

I haven't looked through the code very closely but at high level seems fine to have that from a cursory look.

Hi, @pan3793 and all. Shall we conclude this thread?
Given the discussion, I believe we can close this PR and move forward to generalize --wait for all services (if someone wants to implement that for consistency.)

Thank you so much for making this PR and letting us conclude this topic, @pan3793 !


Avoid crashing on "scale(1)" (short for "scale(1, 1)")

The scale function on the transform attribute has a short form, where
only the first argument is used.  But optimizeTransform would always
assume that there were two when checking for the identity scale.
Closes: #190
Signed-off-by: Niels Thykier<EMAIL_ADDRESS>

Codecov Report

Merging #191 into master will increase coverage by 0.42%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master    #191      +/-   ##
=========================================
+ Coverage   93.08%   93.5%   +0.42%
=========================================
  Files           5       5
  Lines        2140    2140
=========================================
+ Hits         1992    2001       +9
+ Misses        148     139       -9

Impacted Files
Coverage Δ

scour/scour.py
93.93% <100%> (+0.48%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ddb7d8...18e57cd. Read the comment docs.

The "scale(1) == scale(1 1)" assertion is from: https://www.w3.org/TR/SVG11/single-page.html#coords-TransformAttribute
Quote:
scale(<sx> [<sy>]), which specifies a scale operation by sx and sy. If <sy> is not provided, it is assumed to be equal to <sx>.

this looks good and it has test coverage => merged.

Thanks for the merge. :)


Sketch a UI

Let's use these as a rough target for the UI. Sorry for the illegible handwriting. By screen these say:

Welcome! Huh? [Make a did]
Connect to Healthcare [Provider]. [Connect]
Your account: ___ name, ___dob, ___ addr, ___ [phone-icon] [check], [Create Verified Credential]
Wait for SMS [enter-numbers], [Enter Code]
Your Credential: [credential-details], [Save]


SQL files open in text editor instead of the Microsoft sql extention

Type: Bug
if i create a new text file it takes me to the proper place and i can easily run my queries. however if i save my query and try to open it it opens the sql file as a text file and will not let me open another editor
VS Code version: Code 1.87.0 (019f4d1419fbc8219a181fab7892ebccf7ee29a2, 2024-02-27T23:41:44.469Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item
Value

CPUs
11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz (16 x 2304)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: enabled_ondirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okskia_graphite: disabled_offvideo_decode: enabledvideo_encode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
undefined

Memory (System)
31.67GB (18.38GB free)

Process Argv
C:\Users\ljkey\OneDrive - Polk State College\Desktop\Useful things\enrollment\summer enrollment current.sql --crash-reporter-id ef672384-6b06-453e-8377-16021beae1c7

Screen Reader
no

VM
0%

Extensions (19)

Extension
Author (truncated)
Version

copilot
Git
1.171.0

copilot-chat
Git
0.13.0

vscode-pull-request-github
Git
0.82.0

vscode-dotnet-runtime
ms-
2.0.2

data-workspace-vscode
ms-
0.5.0

mssql
ms-
1.22.1

sql-bindings-vscode
ms-
0.4.0

sql-database-projects-vscode
ms-
1.4.1

debugpy
ms-
2024.2.0

python
ms-
2024.2.1

vscode-pylance
ms-
2024.2.2

jupyter
ms-
2024.2.0

jupyter-keymap
ms-
1.1.2

jupyter-renderers
ms-
1.0.17

vscode-jupyter-cell-tags
ms-
0.1.8

vscode-jupyter-slideshow
ms-
0.1.5

oracledevtools
Ora
21.8.0

r-debugger
RDe
0.5.4

r
REd
2.8.2

A/B Experiments
vsliv368:30146709
vspor879:30202332
vspor708:30202333
vspor363:30204092
vswsl492cf:30256860
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:30899288
c4g48928:30535728
azure-dev_surveyone:30548225
962ge761:30959799
pythongtdpath:30769146
welcomedialog:30910333
pythonidxpt:30866567
pythonnoceb:30805159
asynctok:30898717
pythontestfixt:30902429
pythonregdiag2:30936856
pyreplss1:30897532
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
dsvsc016:30899300
dsvsc017:30899301
dsvsc018:30899302
cppperfnew:30973185
d34g3935:30971562
edj9j583:30969295
a89i1917:30973842
e3gdj431:30958358
ccp2r3:30958157
dsvsc020:30976470

/gifPlease

Thanks for reporting this issue! Unfortunately, it's hard for us to understand what issue you're seeing. Please help us out by providing a screen recording showing exactly what isn't working as expected. While we can work with most standard formats, .gif files are preferred as they are displayed inline on GitHub. You may find https://gifcap.dev helpful as a browser-based gif recording tool.
If the issue depends on keyboard input, you can help us by enabling screencast mode for the recording (Developer: Toggle Screencast Mode in the command palette). Lastly, please attach this file via the GitHub web interface as emailed responses will strip files out from the issue.
Happy coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


Torchaudio + tensorflow + CUDA 11.0 = segfault

🐛 Bug
Importing torchaudio after tensorflow-gpu while using CUDA 11.0 causes a segfault. This issue was originally reported in the AugLy repo: https://github.com/facebookresearch/AugLy/issues/28.
To Reproduce
Steps to reproduce the behavior:
Only happens on CUDA 11.0, so we haven't been able to reproduce this error.
import tensorflow
import augly.audio as audaugs

Output:
2021-06-18 18:42:04.241048: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Segmentation fault (core dumped)

Expected behavior
No segfault.
Environment

What commands did you used to install torchaudio (conda/pip/build from source)?

pip install -r torch==1.8.1 torchaudio==0.8.1 (in AugLy's requirements.txt)

PyTorch Version (e.g., 1.0):

OS (e.g., Linux):

How you installed PyTorch (conda, pip, source):

Build command you used (if compiling from source):

Python version:

CUDA/cuDNN version: 11.0

GPU models and configuration:

Any other relevant information:

Additional context
This problem doesn't happen on our (AugLy's maintainers') environments, only on one user's.

Hi @zpapakipos
What PyTorch version is this? torchaudio==0.8 has no CUDA-related code, so it seems that the issue is caused by importing torch, like two CUDA versions are loaded.
Can you verify that it's not PyTorch?

The error is happening to me as I reported in this issue: facebookresearch/AugLy#28
In my environment it can be reproduced with this commands:
python3 -m venv venv
source venv/bin/activate
pip3 install wheel
pip3 install tensorflow-gpu==2.4.1
pip3 install augly
python3 -c "import tensorflow; import augly.audio as audaugs"

Answering your question, the PyTorch version is the version installed by AugLy torch==1.8.1
Doing a:
strace python3 -c "import tensorflow; import augly.audio as audaugs"

The last file opened before the segfault is: python3.8/site-packages/torchaudio/_internal/fft.py
More information about my environment:

SO: Linux Ubuntu 20.04
Python version: 3.8.5
tensorflow version: 2.4.1
CUDA version: 11.0
Nvidia Driver Version: 450.119.03.
GPU: Quadro P5200

Please let me know if you need additional information.
Thank you.

@mcanan Thanks. Can you replace import augly.audio as audaugs with import torch and see that happens?

It doesn't fail:
python3 -c "import tensorflow; import torch"
2021-06-21 10:18:58.630042: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0

In the other case the output is this:
python3 -c "import tensorflow; import augly.audio as audaugs"
2021-06-21 10:18:45.234129: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Segmentation fault (core dumped)

Oh that's interesting. Can you run python -m 'torch.utils.collect_env' and report the output?

Collecting environment information...
PyTorch version: 1.8.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Quadro P5200
Nvidia driver version: 450.119.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1
[pip3] torchaudio==0.8.1
[conda] Could not collect

Do you use CUDA from both TF and PyTorch?
The original issue description says Only happens on CUDA 11.0. (and I am interpreting this as 'it does not happen on CPU-version of PyTorch').
If you do not use PyTorch for DL-related stuff, as a workaround, it might work to replace the CUDA-enabled PyTorch with one without it.

We don't use PyTorch. We use only TF for DL.
We wanted to test the new AugLy library to test the audio augmentations and we installed it. PyTorch is installed during the AugLy installation.
How can we replace the CUDA enabled PyTorch for the other one?
Should't it be done during AugLy installation?

So first uninstall PyTorch and torchaudio pip uninstall torch torchaudio then install the right version of torch and torchaudio.
To install torch, something like pip3 install torch==1.8.1+cpu torchaudio==0.8.1 will work but the best way to find the correct command is to go to https://pytorch.org/get-started/locally/#start-locally and choose the right configuration for you.

I installed the cpu version and I still have the same segfault.
I can reproduce it from scratch following these steps:
python3 -m venv venv
source venv/bin/activate
pip3 install tensorflow-gpu==2.4.1
pip3 install augly
pip3 uninstall torch torchaudio
pip3 install torch==1.8.1+cpu torchaudio==0.8.1 -f https://download.pytorch.org/whl/lts/1.8/torch_lts.html
python3 -c "import tensorflow; import augly.audio as audaugs"

The output is:
python3 -c "import tensorflow; import augly.audio as audaugs"
2021-06-21 14:05:53.719005: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
Segmentation fault (core dumped)

The output of python -m 'torch.utils.collect_env' is:
Collecting environment information...
PyTorch version: 1.8.1+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3

Python version: 3.8 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Quadro P5200
Nvidia driver version: 450.119.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.2
[pip3] torch==1.8.1+cpu
[pip3] torchaudio==0.8.1
[conda] Could not collect

Thank you

Thanks for the report. One more thing, can you try torch==1.9.0 and torchaudio==0.9.0?
It's unlikely that we can make a change to the past release (1.8/0.8), but if we can fix it on master, then we can do a minor release 0.9.1

It doesn't fail now. I followed these steps from scratch:
python3 -m venv venv
source venv/bin/activate
pip3 install tensorflow-gpu==2.4.1
pip3 install augly
pip3 uninstall torch torchaudio
pip3 install torch==1.9.0+cpu torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
python3 -c "import tensorflow; import augly.audio as audaugs"

The only error messages I had were these during torch and torchaudio installation:
ERROR: augly 0.1.1 has requirement torch==1.8.1, but you'll have torch 1.9.0+cpu which is incompatible.
ERROR: augly 0.1.1 has requirement torchaudio==0.8.1, but you'll have torchaudio 0.9.0 which is incompatible.

@mcanan Thanks. Grad that it works.
@zpapakipos @jbitton Not sure if the version requirements should be bumped up in AugLy's requirements.txt, but at least this seems to work.

@mthrok thank you for helping debug this issue! I'll check the unit tests for AugLy audio with torch 1.9.0 and torchaudio 0.9.0 and see if we're still getting the same expected results :)

Closing this issue as it does not happen in recent release (0.9) and master branch.
The reason why it does not work with 0.8 is still unknown but we do not update the past release as-well, so we recommend users to use the 0.9.


accourdion-menu and is_active class

accourdion-menu - options data-multi-open=false

`Tag <li class="is_active">  in .sub-menu <ul> list set display: none like <ul tree>`

Please describe your issue more detailed.

Hi @ivanff, please provide some more info so we can address this issue.
Without any further changes we may close this as it is currently not clear what the issue is.

@ncoden what should we do here? The issue is not clear to me.

Hi @DanielRuf ,
I think i got the same problem with @ivanff here, I can further explain the issue.
When you have data-multi-open set to false, there's some javascript that add display: none to the current menu item.
Hope that helps.
Thanks,

Please provide a working codepen @splitmango-duy

This happens on a wordpress site, using foundation 6.4.3, not sure how I can reproduce the issue with codepen. Because it wont have the class names, such as current-menu-item & current-menu-ancestor attached with wordpress.

Can you provide an URL or copy the relevant html code from theb"view source" contextmenu?

No reply. Closing.


fix: ToC not highlighting properly

This PR should fix ToC not highlighting the correct header.
It also checks if there's any embedded gist markdown with headerLink since it's also affect the ToC.
I don't know what tools you use to generate the minifed version in assets folder so I ignored that for now.
resolve #21

Unfortunately, your solution only corrects the situation for one topic of the article, but creates problems in others.
For example, if you run exampleSite, then on the article
http: // localhost: 1313 / theme-documentation-content
clicking on 3-content-summaries in the TOC will not take you to this section the first time.

Apparently, the introduction of various designs for articles made this script not working, and you need to invent something anew

I've adjusted the fix, please check it.
It's a little bit janky but it should highlight it properly now

Unfortunately, the problem persists

The first time you press the TOC, it does not go where it should

But it seems correct to show where we are.

This is what happens when I click on item 4.1 for the first time.

The second and subsequent times, everything is fine.

And I found another strange feature. For some reason, if the title is not in English, then it does not mark it as active

@noaione I would be very glad if you fix these problems, as I know JS rather poorly.

Yeah, I don't know how to fix that problem currently, but I think that's the (desktop) browser problem not rendering everything yet so the linking is broken.

@noaione Hello again. I found where the cause of the problem was.
$tocLinkElements[i].hash.slice(1) === $selHeader.id
The fact is that if the title is not English, then there will be gibberish on the left side, and the condition is not met
Examples in the screenshot

And I would like you to appreciate our decision, or perhaps with new information would correct your changes.
We have replaced this condition with the following
$tocLinkElements[i].hash.slice(1) === encodeURI($selHeader.id)
And this solved the problem with another language

Adjusted code as requested

@noaione Many thanks for the help


Upgrade black to version ~=22.0

Upgrades the huggingface_hub library quality tool black to use the first stable release of black, version 22.0.

We should also update the setup.py so it trickles into the CI/CD


Bug: Shaders Have Different MD5Sums Between Windows and Linux SDK Builds From Github Runner Builds

Description
I was checking to see differences between the FreeBSD, Windows and Linux builds for #778 to see if something was happening to cause some of the resources to not load properly and decided to just run md5sum on some of the shaders. I chose one that doesn't load on the BSD build for this test.
Prerequisites
Not applicable here.
Steps to Reproduce

Download the Windows and Linux artifacts from here: https://github.com/thunder-engine/thunder/actions/runs/9771559828
Extract each to a separate directory.
Run md5sum release/install-root/sdk/2024.2/resources/engine/materials/DefaultFont.shader from the Windows extraction directory.
Repeat with the Linux extraction directory.

Expected behavior:
Shaders should have the same MD5Sums if they are just copied over and not processed based on platform.
Actual behavior:
The MD5Sums are drastically different.
Reproducibility:
Can repeat with the last stable release builds too.
Versions
Not applicable in this case.
Additional Information
This may not be a bug but an intended outcome. I haven't fully figured out how resources are processed yet. Both the Linux and BSD MD5Sums are exactly the same between builds. If this is a bug, I wonder if this could be related to them not loading correctly.
Attached is a list of my MD5Sum findings for this test. The lines with windows-stable and linux-stable are tests ran on the last stable release build from here. The others are from this PR https://github.com/thunder-engine/thunder/pull/777
md5sum.txt

MD5 check sums was done on top of QT. I'm not controlling this. I can assume these sums was not updated in time when I changed the content. It's not lead to any issues except re import asset if sum doesn't match

I'll close this then. Better safe than sorry in this case.


Spatial - read function and fixes

Update:

Added read_visium function, based on understanding output from 10x
addressed #1027 by changing requirements
added features to import in adata.obs, in addition to coordinates in adata.obsm, in scanpy.datasets

please don’t copy & paste code.
visium_sge needs to be almost empty and relying on read_visium.

should be good now


Typo on microchippic32.json file

What kind of issue is this?

[ ] Question. This issue tracker is not the place for questions. If you want to ask how to do
something, or to understand why something isn't working the way you expect it to, use
our Community Forums https://community.platformio.org

[ ] Feature Request. Start by telling us what problem you’re trying to solve. Often a solution
already exists! Don’t send pull requests to implement new features without first getting our
support. Sometimes we leave features out on purpose to keep the project small.

[ ] PlatformIO IDE. All issues related to PlatformIO IDE should be reported to appropriate repository
https://github.com/platformio/platformio-atom-ide/issues

[X ] Bug report. If you’ve found a bug, please provide information below.

You can erase any parts of this template not applicable to your Issue.

Description of problem
There is a typo error on line 122 of the microchippic32.json file. Where it says cchipKIT-application-32MX250F128.ld it should say chipKIT-application-32MX250F128.ld This prevents compilation on the chipKit-pi board.
Steps to Reproduce

Try to compile anything for the chipkit-pi

Actual Results
[07/18/16 12:16:14] Processing chipkit_pi (platform: microchippic32, board: chipkit_pi, framework: arduino)
pic32-g++ -o .pioenvs\chipkit_pi\firmware.elf -w -Os -mdebugger -mprocessor=32MX250F128B -mno-peripheral-libs -nostartfiles -Wl,--gc-sections -Wl,--script="cchipKIT-application-32MX250F128.ld" -Wl,--script=chipKIT-application-COMMON.ld .pioenvs\chipkit_pi\src\tmp_ino_to.o "-LC:\Users\Maker I.platformio\packages\framework-arduinomicrochippic32\cores\pic32" "-LC:\Users\Maker I.platformio\packages\framework-arduinomicrochippic32\variants\ChipKIT_Pi" "-LC:\Users\Maker I.platformio\packages\ldscripts" -L.pioenvs\chipkit_pi -Wl,--start-group .pioenvs\chipkit_pi\libFrameworkArduinoVariant.a .pioenvs\chipkit_pi\libFrameworkArduino.a -lm .pioenvs\chipkit_pi\libSoftwareSerial.a .pioenvs\chipkit_pi\libAdafruit_VC0706.a .pioenvs\chipkit_pi\libSPI.a .pioenvs\chipkit_pi\libSD.a -Wl,--end-group
pic32-cpp.exe: error: cchipKIT-application-32MX250F128.ld: No such file or directory
pic32-cpp.exe: warning: '-x c' after last input file has no effect
pic32-cpp.exe: fatal error: no input files
compilation terminated.
c:\users\maker i.platformio\packages\toolchain-microchippic32\bin\bin\gcc\pic32mx\4.8.3........\bin/pic32-ld.exe: cannot open linker script file cchipKIT-application-32MX250F128.ld.00: No such file or directory
collect2.exe: error: ld returned 255 exit status
scons: *** [.pioenvs\chipkit_pi\firmware.elf] Error 255
[ ERROR ] Took 6.16 seconds
Expected Results
Compiled file
If problems with PlatformIO Build System:
The content of platformio.ini:
[env:chipkit_pi]
platform = microchippic32
framework = arduino
board = chipkit_pi

Thanks a lot for the report!


Consider setting .spec.ttlSecondsAfterFinished in the Job

Setting .spec.ttlSecondsAfterFinished to some value will automatically clean up and remove the Job and associated Pods (see here for more details).  This way, the Jod-related Pod doesn't linger on in a Completed state when someone lists the pods in a namespace.
I think the change would go somewhere in this method https://github.com/k3s-io/helm-controller/blob/master/pkg/controllers/chart/chart.go#L375.  This might be something added to the kind: HelmChart resource or a global configuration set on the controller or both.

The HelmChart controller does not actually watch the Job to see if it completes. It just periodically re-applies the desired Job spec, assuming that no change means everything is correct. If the Job was cleaned up after it completed, the Helm controller would just re-create it, and it would run again.
In order to allow the Jobs to be cleaned up after they complete, we would need to actually watch the Jobs, ensure that they eventually run to completion, and then store that information somewhere in the HelmChart resource - similar to how the Job controller watches Pods to ensure that they run successfully. This would be a fair bit of additional complexity, and is not currently planned.

Ok.  Closing this issue.


Weather 1 Day Off

I am currently experiencing an issue where the weather is one day off of what is on Environment Canada’s website. For example, the app is showing Saturday’s weather for Sunday, and Sunday’s weather for Monday. The date and time are correct on my phone, and the WeatherCAN app on my phone matches what I see on Environment Canada. I’d really love to be able to use this app as it is gorgeous, so I’m hoping this can be resolved. Thank you in advance!

Update: The issue seems to be gone this morning, so it looks like the issue might only occur when it is late at night.

In case it helps, when I looked last night, it was showing “TODAY” as Saturday (last night was Friday). When I looked this morning, it was still showing “TODAY” as Saturday, which is now correct.

Hey @williamlauzon, thanks for flagging this. This is actually partly due to the data returned from Environment Canada (the WeatherCAN app uses a private internal EC API and not the MSC Datamart as Clouds/Nimbus use, so the private API may already have this fixed). In rare occasions, the days returned from the Datamart are mislabelled and are offset by 1 day. There is also, unfortunately, no good way of parsing the data since only weekday names are used in the raw data as opposed to timestamps.
That being said, the actual logic to parse the dates occurs in lfroms/nimbus, so I'll create a corresponding issue there. I'll keep an eye out for the next occurrence and confirm the logic.

Hi @lfroms, thanks so much for the quick reply! I really appreciate you looking in to this and the effort you've put into making this great app! It's too bad that the issue is on the MSC Datamart side, but hopefully it will be fixed by them at some point. Again, thank you for your work!


Identity Server 4.1.2 upgrade

Included

Upgrade to target both .Net Core 3.1 and .Net 5
Upgrade to latest IS packages
Upgrade general packages and projects (Except VueJs project)
Configure an existing "free" to use OpenLdap in the test project, so that it will be easier to simply kick start a demo
Small update to the Readme

What is NOT included in this change

Docker image update in any ways, so build might simply fails
Usage of git hooks to build automagically the whole shabang.

Fix #65


Empty group put request 500s

Fixes #843
Review Checklist

Tests were added to cover all code changes
Documentation was added / updated
Code and tests follow standards in CONTRIBUTING.md

An empty payload is an improper request, so the more accurate response code would be 400: Bad Request. I would add a check to group/project/session/acquisition/collection/user PUT endpoints that checks to make sure the payload is not empty as soon as possible in the handler logic.

It should really be on all PUT endpoints, feel free to add if you'd like, otherwise that list above is a good start.

Looks good after response message is changed to be a bit more descriptive!

LGTM!

Thanks Harsha.


support with

Current dependencies on/for this PR:

main

PR #64   👈

This stack of pull requests is managed by Graphite.


Tab completion in REPL

Closes #1182. This is based on the changes in PR #2155.

@brson, please review this PR when you have a chance. thanks!

@raghunandanbhat Looking at it now.

FYI we need to not introduce any new dynamic allocation to the repl (we can allocate during init, but not during execution - that's tigerstyle), so I am attempting to (quickly...) remove the dynamic allocation from https://github.com/tigerbeetle/tigerbeetle/pull/2155 to get that landed and unblock you @raghunandanbhat. It will almost certainly be a merge conflict for you.
My work on that is https://github.com/brson/tigerbeetle/tree/interactive-repl-improvements-static. I think I can submit it today, and it will require another round of review from another tigerbeetle team member.
In the meantime we can still make progress on the review here.

Thanks for the feedback @brson.
I have already made some progress on making the code to use static allocations, based on the changes in your branch. I will have it ready for review once your PR lands.

This PR adds new local variables to the read_until_newline_or_eof_alloc function, which is already probably the densest function in the repl, and after removing dynamic allocations, those ArrayLists will end up being members of ReplType. It seems like we might be able to encapsulate all the completion state into another struct, perhaps moving some of the completion logic out of the big read_until_newline... function into methods on a completion type. Please give that some thought and see if it makes sense to you. I'm not determined to make this change now, but I think it could be a readability improvement.

I’ll look into moving the completion logic into a separate struct. My main concern is that much of the code is focused on cursor position adjustments, so I’m unsure how much readability will improve. I’ll see if there’s a way to balance clarity without adding extra back-and-forth for readers trying to follow the logic.

In testing, I do see some bugs in how things are displayed in the terminal, particularly in corner cases where the input spans multiple lines. Unfortunately, I'm not prepared to describe them in detail right now, but I definitely will.

I have seen this bug when the input overflows to next line and I think I have fixed it. Would still like to hear your feedback once you have the chance to test it.

@brson - I made changes after your review, please review and let me know your feedback.
I moved completion state to a new Completion struct with static allocations -

a RingBuffer to hold  256 completion entries, each entry of length 512.
stdx.BoundedArray for the three buffers- prefix, query and suffix.

In testing, I do see some bugs in how things are displayed in the terminal, particularly in corner cases where the input spans multiple lines. Unfortunately, I'm not prepared to describe them in detail right now, but I definitely will.

I've fixed this bug. Please retest this part and let me know if you see any issues.
Also, if you have any suggestion to make the ANSI escape codes more readable, please let me know.

The prerequisite pr has now been merged: https://github.com/tigerbeetle/tigerbeetle/pull/2464
Thanks for your patience.

@brson please review the changes when you have a chance. thank you.

Some of the long-line artifacts are resolved now, but I still find one obvious one when the terminal width is smaller than the completion menu:

Some of the long-line artifacts are resolved now, but I still find one obvious one when the terminal width is smaller than the completion menu:

could you please tell me if you're resizing the terminal window in the middle of command entry while window size is smaller than completion menu. If so, this is expected because terminal size is refreshed only after ctrl+c/newline, not for every keystroke. This was also mentioned in the REPL enhancement PR and Python repl also behaves the same.
If you resize the window, then hit \n and then completions menu toggle should work. Please provide steps to reproduce this if you still see this issue.

could you please tell me if you're resizing the terminal window in the middle of command entry while window size is smaller than completion menu. If so, this is expected because terminal size is refreshed only after ctrl+c/newline, not for every keystroke. This was also mentioned in the REPL enhancement PR and Python repl also behaves the same.
If you resize the window, then hit \n and then completions menu toggle should work. Please provide steps to reproduce this if you still see this issue.

ok, yes it is as you describe - from an empty prompt i am resizing the window, then typing "c-tab-tab-tab" etc produces artifacts. surprising, but this is because the window size is calculated when the prompt is first displayed, then i resized the window.
So I agree there's nothing to fix about that, at least for now, since python etc. has similar behavior wrt window resizing.

Looks great. Thanks @raghunandanbhat!


NuoDB storage manager rescheduled pod is taking ~12 minutes to come to running state.

NuoDB storage manager rescheduled pod is taking ~12 minutes to come to running state.
Steps Performed:

Deployed NuoDB with 3 Admin, 2 SM (consuming cstor volume of size 800G) and 3 TE configuration on OpenShift 3.10 EE.
Ran YCSB load for 8 days.
Performed Storage Manager pod kill, observed that the rescheduled application pod took around 12 minutes to come to running state.

Describe events showed that application pod stuck at mount/umount.

Please find the attached target and application logs.

target.log
sm.log

Application pod describe events showed
mount/umount stuck around 8 minutes