Adding  tag support

Greetings,
As Bootstrap v4 replaces the old .progress <div> with a real <progress> element, we should consider adding <meter> support to better match the HTML semantics of each element.
I would also suggest that <progress> tag should be stripped by default, or even animated since it aims to represent the completion progress of a task (quoting WHATWG spec).

I would also suggest that  <progress>  tag should be stripped by default, or even animated

Different folks always want different aesthetics. Plain-by-default with a single class each to opt into striping and animation makes it easy to keep everyone happy. Using 2 more classes isn't a heavy burden.

Closing as a won't fix given the .progress and lack of IE10 or IE11 support.

Could this be reconsidered for v5?

Could this be reconsidered for v5?

I suppose so! How would meter differ from progress overall? Anything meaningful in it's display or rendering?

I suppose so! How would meter differ from progress overall? Anything meaningful in it's display or rendering?

As @aduh95 already explained:

stripped by default, or even animated since it aims to represent the completion progress of a task

But @cvrebert already argued that it would be better to stick with plain styling by default and allow users to add additional striping and animation classes. I tend to agree with that.

As @aduh95 already explained:

stripped by default, or even animated since it aims to represent the completion progress of a task

But @cvrebert already argued that it would be better to stick with plain styling by default and allow users to add additional striping and animation classes. I tend to agree with that.

Yeah, that still holds true IMO. Animating by default can be a performance hit for sure.

Yeah, that still holds true IMO. Animating by default can be a performance hit for sure.

I just realized that <progress> itself hasn't been supported since #21473. @mdo Do the reasons for removing <progress> sill apply or has something changed? I guess those reasons are likely to also affect <meter>.

I just realized that <progress> itself hasn't been supported since #21473. @mdo Do the reasons for removing <progress> sill apply or has something changed? I guess those reasons are likely to also affect <meter>.

Yeah still applies as far as I know. Closing out, but if someone finds a solution, I'll happily revisit :).


NIFI-12219 Add Xodus Flow History Storage with Migration

Summary
NIFI-12219 Backports Flow Configuration History persistent storage using JetBrains Xodus as implemented for NIFI-12206 and adds migration capabilities to the support branch for subsequent NiFi version 1 releases.
The EntityStoreAuditService class contains the same implementation as merged for NIFI-12219, and the new FlowConfigurationHistoryMigrator supports automated migration from H2 to Xodus on startup. The migrator checks for the presence of the H2 database file named nifi-flow-audit.mv.db and proceeds with migration when found. The migration process includes checking for older H2 binary formats, supporting all previous versions of H2 from 1.4 through 2.2. The migration process then uses available paging capabilities from the H2-based AuditService to transfer records. Following successful migration, the FlowConfigurationHistoryMigrator renames nifi-flow-audit.mv.db with a backup timestamp extension.
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[X] Apache NiFi Jira issue created

Pull Request Tracking

[X] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[X] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[X] Pull Request based on current revision of the main branch
[X] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[ ] Build completed using mvn clean install -P contrib-check

[ ] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

Thanks for putting this together, @exceptionfactory! Code looks good. Did a quick test and everything appears to have been migrated properly. Able to see all of the events that I expected, including before/after values, timestamps, action taken, user... +1 will merge to support/nifi-1.x!


GET, PUT commands do not work for device-mqtt

🐞 Bug Report
Affected Services

 The issue is located in: device-mqtt, while performing get, set operations via edgex UI localhost:4000, 5XX error response is reported for performing get or set, additionally the ping operation from CLI does not return 200 response:
mpunix@LIN-MP22QN1X:~/edgex-foundry/edgex-compose-3.1.1-mqtt$ curl http://localhost:59982/api/v3/ping | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    90  100    90    0     0   7930      0 --:--:-- --:--:-- --:--:--  9000
{
   "apiVersion" : "v3",
   "serviceName" : "device-mqtt",
   "timestamp" : "Mon Jul 15 00:35:52 IST 2024"
}

Is this a regression?

Not sure because I have not worked on previous versions.
Description and Minimal Reproduction

Go to compose builder & perform the following command make gen ds-mqtt mqtt-bus
Run the generated file.
go to localhost:4000 and try giving commands to the device.
5XX error is reported.

🔥 Exception or Error
device-mqtt logs:
edgex-device-mqtt  | level=INFO ts=2024-07-14T18:51:01.791685687Z app=device-mqtt source=message.go:51 msg="Starting device-mqtt 3.1.1 "
edgex-device-mqtt  | level=INFO ts=2024-07-14T18:51:01.791690179Z app=device-mqtt source=message.go:55 msg="device mqtt started"
edgex-device-mqtt  | level=INFO ts=2024-07-14T18:51:01.791694734Z app=device-mqtt source=message.go:58 msg="Service started in: 97.544869ms"
edgex-device-mqtt  | level=ERROR ts=2024-07-14T18:55:11.753606883Z app=device-mqtt source=driver.go:111 msg="Handle read commands failed: can not fetch command response: method=get source=randfloat64"
edgex-device-mqtt  | level=ERROR ts=2024-07-14T18:55:11.753811405Z app=device-mqtt source=restrouter.go:159 X-Correlation-ID=08402bf3-41a3-4eda-8807-8d87492f97bf msg="error reading Regex DeviceResource(s) randfloat64 for MQTT-test-device -> can not fetch command response: method=get source=randfloat64"
edgex-device-mqtt  | level=ERROR ts=2024-07-14T18:55:31.822734925Z app=device-mqtt source=driver.go:177 msg="Handle write commands failed: can not fetch command response: method=set cmd=randfloat64"
edgex-device-mqtt  | level=ERROR ts=2024-07-14T18:55:31.824457374Z app=device-mqtt source=restrouter.go:159 X-Correlation-ID=f836236b-c427-4bd5-a2d9-92604c6d0eb9 msg="error writing DeviceResource randfloat64 for MQTT-test-device -> can not fetch command response: method=set cmd=randfloat64"
edgex-device-mqtt  | level=ERROR ts=2024-07-14T18:56:08.748848372Z app=device-mqtt source=driver.go:111 msg="Handle read commands failed: can not fetch command response: method=get source=ping"
edgex-device-mqtt  | level=ERROR ts=2024-07-14T18:56:08.748967647Z app=device-mqtt source=restrouter.go:159 X-Correlation-ID=241790d0-3b4c-45a3-9654-dabbc79ebf73 msg="error reading Regex DeviceResource(s) ping for MQTT-test-device -> can not fetch command response: method=get source=ping"

🌍  Your Environment
Deployment Environment: Windows 11, WSL2 - Ubuntu 22.04
EdgeX Version: 3.1.1
Anything else relevant?
No.

Thanks

Hi @M0hanrajp, these questions are not EdgeX bug, they are common usage issues, maybe you should move to the discussion channel.
It's not a good idea if you want to test EdgeX in the WSL. You need a pure Linux system like Ubuntu version 22 or 23.

you have to config the mqtt broker properly, because the default mqtt broker of device-mqtt is not the internal mqtt bus
https://github.com/edgexfoundry/device-mqtt-go/blob/35de5baf133cb3849b13998f5bd615ef0c638656/cmd/res/configuration.yaml#L30
Also, you need to set up the MQTT simulator on your own, or nothing will send the response to the MQTT response topic.

@jiekechoo & @cloudxxx8
Thanks for both of your response.


I founded a bug on vuejs forum, what can i do?

The bug is related with the verification email and the github authentication.
Where can i report or try to solve it?

You can share more details here if that's ok. The forum is using NodeBB and its GitHub social plugin.

Ok, i'll explain, it's quite a bit simple...
I've registered my account using the github connector and i received a warning saying that i need validate my email address, but the confirmation never reached my email box.
I checked my registered e-mail on the "Profile Edit" page in the Forum application and realized that my email was registered using a strange account from github, something like<EMAIL_ADDRESS>For some reason it failed to obtain the email address from my github account . The reason could be the privacy settings from the github, cause i had the email configured as private at the time of registration.
After i manually updated my email address, i was able to receive the confirmation. Anyway, this needs more tests to confirm the real cause of the problem.

I just wanted to create an issue for that. It didn't come to my mind that the email address would be something else then my own one.

@nerdyglasses, my privacy settings on github was setted as "private", so my e-mail was hidden. I have almost sure this is the cause.
It's not a problem with vue site itself, this problem came from the GitHub social plugin. Here's one related issue from their repo: https://github.com/julianlam/nodebb-plugin-sso-github/issues/4

I'm glad that we've documented a solution here, but I'll close for now, since there's nothing we can do to fix it ourselves. 😕


the 5th question's solution doesn't work.

%run python -c "import numpy; numpy.info(numpy.add)"
It yields:
ERROR:root:File `u'`python.py'` not found.

Weird. How do you start a regular python session from the terminal?

just type the 'python' command in cmd.exe

in the cmd.exe, this line work:
python -c "import numpy; numpy.info(numpy.add)"

I'm wondering why the %run command doesn't recognize this command line as a line of python code.

Oh, the line is supposed to be python -c "import numpy; numpy.info(numpy.add)" without the run actually.

You mean in the jupyter notebook?

No, on the terminal.

ok, yet don't know how to run it in jupyter notebook. Thanks for your patience anyway.

From within Jupyter (or python session):
import numpy
numpy.info("add")

thank you! maybe the answer doesn't work for me.

I think %run is wrong. You should replace %run with "!" if you want to run a shell command within Jupyter, which may be useless.

also this problem,  @mgdreamingstar
Input :
%run `python -c "import numpy; numpy.info(numpy.add)"`

Error :
ERROR:root:File `'`python.py'` not found.

the IPython built-in magic command %run can't recognize python as a program (because it automatically looks for 'python.py'), so you can change this into ! which works for me.
!python -c "import numpy; numpy.info(numpy.add)"


Issue with BeforeAppendModel when selecting columns to update

Model
type cameraBM struct {
	bun.BaseModel bun:"table:camera"
	ID            uuid.UUID `bun:"type:uuid,default:uuid_generate_v4(),pk"`
	CreatedAt     time.Time `bun:",nullzero,notnull,default:current_timestamp"`
	UpdatedAt     time.Time `bun:",nullzero,notnull,default:current_timestamp"`
	Config        json.RawMessage `bun:"type:jsonb"`
}

Hook
var _ bun.BeforeAppendModelHook = (*CameraBM)(nil)

func (u *CameraBM) BeforeAppendModel(ctx context.Context, query bun.Query) error {
	switch query.(type) {
	case *bun.InsertQuery:
		u.CreatedAt = time.Now()
	case *bun.UpdateQuery:
		u.UpdatedAt = time.Now()
	}
	return nil
}

Update Query which updates updated_at
_, err = db.NewUpdate().
			Model(&cameraBM).
			Column("config", "updated_at").
			WherePK().
			Exec(ctx)

Update Query which I expect to update updated_at
_, err = db.NewUpdate().
			Model(&cameraBM).
			Column("config").
			WherePK().
			Exec(ctx)

I am not sure if this behaviour is expected or not, I spent hours trying to figure out why the hook wasn't working, maybe the hook should add the appropriate column if it doesn't already exist.
DB used is Postgres 15

@rohanramaswamy
The SQL resulting from the query build in the first example looks like this
UPDATE camera SET config = "...", updated_at = "..." WHERE ID = "...";

The SET clause specifies the updated_at column, so it is updated.
On the other hand, the SQL resulting from the query build in the second example looks like this
UPDATE camera SET config = "..." WHERE ID = "...";

The SET clause does not specify the updated_at column, so it is not updated.
I think the behaviour is as expected because the responsibility of BeforeAppendModel method should only be to update the model field (in this case CreatedAt or UpdatedAt), and the responsibility of the Column method should be to determine which column values are actually updated, and those two responsibilities should be separated.


Incorrect old balances in computeDeltas

Describe the bug
The computeDeltas() function has an old balances and new balances params but right now we always put same justified balances of store to both of them
(https://github.com/ChainSafe/lodestar/blob/v0.39.0/packages/fork-choice/src/forkChoice/forkChoice.ts#L190)
Expected behavior
We should keep the own balances (as old balances, what reflect the weights in ProtoArray) somewhere, and forkchoice store should give new justified balances to computeDeltas(). The implication is we want to reflect the weight of votes if effective balances of validators get changed.
Refer to lighthouse for a reference implementation.

Right! Notice this also when working on this issue. What are the implications of always passing the same pair of new and old balances?

Right! Notice this also when working on this issue. What are the implications of always passing the same pair of new and old balances?

the impact is not much because effective balances of a validator are rarely updated per epoch, we also passed forkchoice spec tests for a while

Right! Notice this also when working on this issue. What are the implications of always passing the same pair of new and old balances?

the impact is not much because effective balances of a validator are rarely updated per epoch, we also passed forkchoice spec tests for a while

It's okay to devote most efforts on making Lodestar perform well on good network conditions. But it's equally important to make sure it works correctly under adverse network conditions. Differences in fork-choice logic between clients can be grounds for attack, so everyone must follow the same spec to ensure a safe network


open to <IP_ADDRESS> when host is <IP_ADDRESS>

When using address <IP_ADDRESS>, open directly to <IP_ADDRESS>. Fixes an issue where Windows doesn't translate this address correctly.
fixes #727

Fix verified in #727


Move iOS native libraries to iOS project

Moves the iOS BASS libraries out of osu.Framework.NativeLibs and into osu.Framework.iOS.

Reduces the size of the NativeLibs package.
Increases the size of the iOS package.
We will no longer be distributing iOS native libraries with osu! desktop, as they were being brought in with the NativeLibs package.  This is negligible at the moment, but will become apparent when the FFmpeg libraries are added, as they are quite large (138MB).
The iOS native libraries will be copied into the output directory when building Xamarin projects, so no need to duplicate them in the osu! repository.

There is a small caveat.  In the osu.iOS project, since the libraries are extracted into the output directory on build, the NativeReference items need to point directly to $(OutputPath) to tell mtouch to copy them into its cache for native linking.
The catch is that the files are not extracted to the build directory until AFTER mtouch has cached the native libraries.  This means that the first build after a clean will fail, and you will need to rebuild such that mtouch can cache the libraries.
Merging this PR will not cause any problems with the existing setup, as we currently store duplicates of the native libraries in the osu! repository.  This will only become a problem when we remove those duplicates and rely solely on the libraries from osu.Framework.iOS.

Note that this PR will likely be a requirement for using FFmpeg on mobile.  Once this PR is merged I will have more PRs incoming to add the FFmpeg libraries, and to update the osu! iOS project to remove the duplicate libraries and point to the ones from the osu.Framework.iOS package.  We will also need to do the same for Android.

Also keep in mind that macOS currently doesn’t have bundled ffmpeg and relies on it being installed on the host.

They'll be different libraries to the iOS ones though, so if we decide to distribute FFmpeg with desktop for macOS they will just go in osu.Framework.NativeLibs no problem.  The iOS project won't import them since ExcludeAssets="all" is specified.

Updated to indicate new findings in regards to the native reference wildcard.

Alright, this is good to go.  After this I'll be making a PR to the framework repo to add the FFmpeg libraries, and a PR to the osu! repo to remove duplicate libraries and reference FFmpeg.


Clean up the SSE client side transport

Renamed StreamPipeConnection to PipeReaderFactory
Flow the transport cancellation token to the CopyToAsync routine ~and use it in ReadAsync (as much as it pains me)~
Use the PipeWriterStream from src/Common
Other small cleanup and nits to make the style consistent with the other pipe reader loops

🆙 📅


Updating gym to gymnasium and cleaning some imports

Hello,
I am working a lot with Sofa in the last few years for my PhD and I have found this repository as @ScheiklP mentioned one code snapshot from it when I was struggling with a grasping simulation. I have to say that this repository is really impressive and I would be really interested to use some of the environments to train some DNN for my PhD and add some extra functionalities along with my work.
However, I came across some problem when trying to install the python package on my computer. I found the error was coming from the Gym package. As I could read at the official repository from OpenAI about Gym, they stopped maintainance on this repository on 2021. The development was moved to Gymnasium and now the recommended form of use is import gymnasium as gym. I think it will be good for new users to have this updated on the main branch. Do you want me to submit a pull request with the necessary changes?
Regards,
Mikel

Hi @mikelitu,
thanks a lot for your kind words! :)
I actually have a local branch where I already updated to gymnasium.
I just did not have the time yet to properly review and push it.
I will try to do that in the coming weeks and would be very grateful, if you could review the PR. Would you be interested in that?
Cheers,
Paul

Hi @ScheiklP,
Thank you for the fast respond. Sure, happy to review the PR once you are done with it.
Best,
Mikel

Hi @mikelitu,
from a first check, that went easier than expected.
Could you head over to the PR and give it a test drive?
Cheers,
Paul


Improve documentation for browser.enterRepl

Follow up to What is the difference between pause() and enterRepl()?.
Protractor API docs for pause() and enterRepl() are very similar and can cause confusion and questions about differences between the two.
Also, Debugging page should probably have a note about enterRepl().
Thanks!

We should probably just make enterRepl private - I don't think there's a reason for users to use anything other than pause.

This is also relevant to the recent debugger changes.

I think that this is taken care of in https://github.com/angular/protractor/pull/3828


Reviewer 1 - errors using the example code

openOmics version: -
Python version: 3.9
Operating System: Linux

Description
The following code was available in an earlier version of the README (now https://openomics.readthedocs.io/en/latest/) and contained a few incorrect points:
'# Load each expression dataframe': additional ')' in lines should be removed! as current form results in an error
mRNA = MessengerRNA(data=folder_path+"LUAD__geneExp.txt", transpose=True, usecols="GeneSymbol|TCGA", gene_index="GeneSymbol", gene_level="gene_name")
Results in warnings (only in first time use):
'/homes/.local/lib/python3.7/site-packages/openomics/transcriptomics.py:95: FutureWarning:
read_table is deprecated, use read_csv instead.
/homes/.local/lib/python3.7/site-packages/openomics/transcriptomics.py:95: ParserWarning:
Falling back to the 'python' engine because the 'c' engine does not support sep=None with delim_whitespace=False; you can avoid this warning by specifying engine='python'.'
som = SomaticMutation(data=folder_path+"LUAD__somaticMutation_geneLevel.txt", transpose=True, usecols="GeneSymbol|TCGA", gene_index="gene_name")
Results in: 'KeyError: 'gene_name''. This should probably be 'gene_index="GeneSymbol"'.
luad_data.add_clinical_data(clinical_data=folder_path+"nationwidechildrens.org_clinical_patient_luad.txt")
Results in warning: '/homes/.local/lib/python3.7/site-packages/openomics/clinical.py:51: FutureWarning:
read_table is deprecated, use read_csv instead, passing sep='\t'.'
gencode = GENCODE(path="ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_32/",file_resources={"long_noncoding_RNAs.gtf": "gencode.v32.long_noncoding_RNAs.gtf.gz","basic.annotation.gtf": "gencode.v32.basic.annotation.gtf.gz","lncRNA_transcripts.fa": "gencode.v32.lncRNA_transcripts.fa.gz","transcripts.fa": "gencode.v32.transcripts.fa.gz"},remove_version_num=True,npartitions=5)
Results in: _'AttributeError: 'io.TextIOWrapper' object has no attribute 'startswith''.

This works now too by the way. I just added this as separate issue so that all my reviewer comments are addressed.


is the preset of the map generator missing?

"strandedblock" is not in the list

have you enabled the mod and restarted?

the language has been set to "some"(idk) language to test for any missing locales

for me stranded block appears


Spark-3.1.2-Azure/Synapse-SQL-Almost Works !

Discussed in https://github.com/OpenLineage/OpenLineage/discussions/653

Originally posted by Grudra1 April  5, 2022
I was trying to get the lineage for a sample job which reads and writes data from Synapse Sql. I know it is not supported for Saprk lineage yet, but since in the documentation, https://github.com/microsoft/sql-spark-connector,
it says :
Supported Options
The Apache Spark Connector for SQL Server and Azure SQL supports the options defined here: SQL DataSource JDBC
which I take as a hint it is built on top of spark jdbc connector and Spark integration supports jdbc connectors for datasets , I gave it a shot.
Well the result is interesting, because it can put the lineage for reads from Synapse, but while I tried to write the dataframe into a Synapse table, it breaks just while trying to post it on Marquez. The event is properly created by the openlineage integration, just the post fails. Error trace is as given below :
22/04/05 11:04:48 ERROR OpenLineageContext: Could not emit lineage:
{"eventType":"COMPLETE","eventTime":"2022-04-05T11:24:34.411Z","run":{"runId":"76491cd9-9abd-457c-9068-250999993c9d","facets":{"spark.logicalPlan":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet","plan":[{"class":"org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand","num-children":0,"query":[{"class":"org.apache.spark.sql.catalyst.plans.logical.Project","num-children":1,"projectList":[[{"class":"org.apache.spark.sql.catalyst.expressions.Alias","num-children":1,"child":0,"name":"VENDORID","exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":19,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[],"explicitMetadata":{},"nonInheritableMetadataKeys":[]},{"class":"org.apache.spark.sql.catalyst.expressions.Cast","num-children":1,"child":0,"dataType":"integer","timeZoneId":"Etc/UTC"},{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"VENDORID","dataType":"string","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":0,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"TOTALPASSENGER","dataType":"integer","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":1,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"TOTALFARE","dataType":"double","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":2,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[]}]],"child":0},{"class":"org.apache.spark.sql.execution.datasources.LogicalRelation","num-children":0,"relation":null,"output":[[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"VENDORID","dataType":"string","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":0,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"TOTALPASSENGER","dataType":"integer","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":1,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[]}],[{"class":"org.apache.spark.sql.catalyst.expressions.AttributeReference","num-children":0,"name":"TOTALFARE","dataType":"double","nullable":true,"metadata":{},"exprId":{"product-class":"org.apache.spark.sql.catalyst.expressions.ExprId","id":2,"jvmId":"99d99999-9f9f-9afd-99e9-9999c97a9999"},"qualifier":[]}]],"isStreaming":false}],"dataSource":null,"options":null,"mode":null}]},"spark_version":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunFacet","spark-version":"3.1.2","openlineage-spark-version":"0.4.0"}}},"job":{"namespace":"spark_integration","name":"test_synapse_py.execute_save_into_data_source_command"},"inputs":[{"namespace":"sqlserver://synapse-test-test001.sql.azuresynapse.net","name":"dbo.TEST_SUMMARY","facets":{"dataSource":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet","name":"sqlserver://synapse-test-test001.sql.azuresynapse.net","uri":"sqlserver://synapse-test-test001.sql.azuresynapse.net"},"schema":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet","fields":[{"name":"VENDORID","type":"string"},{"name":"TOTALPASSENGER","type":"integer"},{"name":"TOTALFARE","type":"double"}]}}}],"outputs":[{"namespace":"sqlserver://synapse-test-test001.sql.azuresynapse.net;databaseName=TESTPOOL1;","name":"dbo.TEST_SUMMARY_COPY_SPARK","facets":{"dataSource":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/DatasourceDatasetFacet.json#/$defs/DatasourceDatasetFacet","name":"sqlserver://synapse-test-test001.sql.azuresynapse.net;databaseName=TESTPOOL1;","uri":"sqlserver://synapse-test-test001.sql.azuresynapse.net;databaseName=TESTPOOL1;"},"schema":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet","fields":[{"name":"VENDORID","type":"integer"},{"name":"TOTALPASSENGER","type":"integer"},{"name":"TOTALFARE","type":"double"}]}},"outputFacets":{"outputStatistics":{"_producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","_schemaURL":"https://openlineage.io/spec/facets/1-0-0/OutputStatisticsOutputDatasetFacet.json#/$defs/OutputStatisticsOutputDatasetFacet","rowCount":0,"size":0}}}],"producer":"https://github.com/OpenLineage/OpenLineage/tree/0.4.0/integration/spark","schemaURL":"https://openlineage.io/spec/1-0-2/OpenLineage.json#/$defs/RunEvent"}
OpenLineageHttpException(code=null, message=

Error 400 Bad Request

HTTP ERROR 400 Bad Request

URI:/api/v1/lineage
STATUS:400
MESSAGE:Bad Request
SERVLET:jersey

, details=null)
        at io.openlineage.spark.agent.OpenLineageContext.emit(OpenLineageContext.java:65)
        at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.endEvent(SparkSQLExecutionContext.java:246)
        at io.openlineage.spark.agent.lifecycle.SparkSQLExecutionContext.end(SparkSQLExecutionContext.java:113)
        at io.openlineage.spark.agent.OpenLineageSparkListener.onJobEnd(OpenLineageSparkListener.java:180)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:39)
        at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
        at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
        at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
        at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
        at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
        at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
        at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
        at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1381)
        at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
As you can see, the run event is perfectly created. All inputs ,outputs, schema..everything is fine. Openlineage did create the lineage which should be good to be posted in Marquez, yet it fails to emit the event.
Is there any quick fix available for this issue ?

As found in the original discussion, the issue is Openlineage is trying to post an event which is perfect from Openlineage standpoint but Marquez cannot handle it. There is a gap between Marquez and Openlineage at this point.

@Grudra1 created Marquez PR that should fix this: https://github.com/MarquezProject/marquez/pull/1936

Thank You @mobuchowski !


Duplicate participants if a participant leaves & joins

In the Unity SDK: BeamManager.SingletonInstance.Participants.Count does not remove a viewer from the list when they leave the channel, but will add them again if they come back to the channel creating a duplicate entry. If the app is restarted, the list returns to the correct number of viewers. Any advice would be helpful.

We've fix this issue with the latest release of the plugin:
https://github.com/WatchBeam/interactive-unity-plugin/releases/tag/v0.7.2-beta


[Bug]: "Repositories / <repo_name>" flickers in GUI

What happened?
Current Behavior:
What actually happened, including error codes if applies.
The repository name ("sample-repo" in the screenshot) on the GUI flickers to "#" and back to the repo name on every tab update.

Steps to Reproduce:

Open a repository in the GUI.
Look at the repo name at the top left, click "Settings"
Watch the repo name flicker to "#" and back to the original repository name.

Expected Behavior
Expected Behavior:
If I am not moving between repositories, there is no need to flicker the repository name.
lakeFS Version
v0.99.0
Deplyoment
Cloud (staging)
Affected Clients
No response
Relevant logs output
No response
Contact Details
@arielshaqed I literally sit right next to you

Still happens.


[Feature] Upload Custom Dictionaries

Users should be able to add and remove custom dictionary files which contain client specific words such as location, sports teams, etc.
Users may/may not be provided write/delete access to the primary wordlists. You can also put a limit on the file size of the wordlist.

There is some code in hashcat and john right now to let you a custom list to the front of a dictionary, but it is comments out and has not been tested. Would this be sufficent? You could have an empty dictionary choice then let people create a custom wordlist to add to it. What is a sane size limit here?

You can now add custom values to the existing list of dictionaries in the 3.10 plugin.


Release 0.5.0

Rendered
Checklist for testing alpha releases:

Windows

[ ] Python Wheel

[ ] Web
[ ] Native

[ ] Rust crate

[ ] Web
[ ] Native

[ ] Rust install

[ ] Web
[ ] Native

Linux (@teh-cmc)

[ ] Python Wheel

[ ] Web
[ ] Native

[ ] Rust crate

[ ] Web
[ ] Native

[ ] Rust install

[ ] Web
[ ] Native

Mac

[x] Python Wheel

[x] Web
[x] Native

[ ] Rust crate

[ ] Web
[ ] Native

[ ] Rust install

[ ] Web
[ ] Native

TODO:

[ ] tagged links
[ ] video
[x] web docs for segment anything would be nice so we can point somewhere
[ ] update this PR with the intro contents once everything's ready
[x] #1931

Considered non blocking :white_check_mark:

[ ] firefox linux web refresh issue? extremely broken?
[x] just py-run-all doesn't actually run all??
[ ] doc missing about how one is suppposed to publish the alphas
[x] multiprocessing example blocks everything
[x] camera example blocks everything
[ ] #1928
[ ] #1929
[ ] #1930
[x] #1932

Considered non blocking :white_check_mark:

[x] just py-run-all...

[x] native
[x] web
[x] save & reload

[x] just rs-run-all

[x] native

Two more PRs that I think are worth cherry-picking over. These are both fairly limited python-only changes and so seem reasonably low risk as long as someone verifies #1939 on Mac.

https://github.com/rerun-io/rerun/pull/1939
https://github.com/rerun-io/rerun/pull/1925


[BUG] PMProgressHandler not emitting events and called even for non iCloud files

Describe the bug
I am fetching videos and, as you have taken the time to explain me in #661, the AssetEntity.file or .originFile can take time to be retrieved on iOS because is not locallyAvailable and needs to be fetched from iCloud.
I have 2 issues with this :

First problem : assetEntity.file or any other method is sometimes fetching the file but i have iCloud disabled on my iPhone and never put any files there, tested on 3 different iPhones without iCloud and all have the download but not all files.

Second problem : In photo_manager: ^1.3.10, i had the progress as below :

File? tempFile;
      if (assetEntity.type == AssetType.video) {
        PMProgressHandler handler = PMProgressHandler();
        StreamSubscription subscription = handler.stream.listen((event) {
          if (kDebugMode) {
            LogManager.w("Event : ${event.state}, progress : ${event.progress} & time : ${DateTime.now().second}");
          }
        });
        tempFile = await assetEntity.loadFile(isOrigin: false, progressHandler: handler);
        subscription.cancel();
      }
      else {
        tempFile = await assetEntity.loadFile(isOrigin: false);
      }

But in 2.0.0-dev.2, i have no events emitted except this one from the above log :
flutter: Event : PMRequestState.prepare, progress : 0.0 & time : 20

And the file is ready 5-10 seconds later.
I am aware that 2.0.0 is still under development of course or maybe it's something i am doing wrong with the methods to fetch files.
To Reproduce
Steps to reproduce the behavior:

Filter your gallery with :  final albums = await PhotoManager.getAssetPathList(onlyAll: true, type: RequestType.video)
Attach the above code as listener
Check if you have not iCloud if some files are anyway downloading
See that no progress is emitted but the load takes few seconds

Expected behavior

The file to not be downloaded when iCloud is disabled on the iPhone (in my case i have never put anything in iCloud so it's not even a user who has disabled iCloud after putting some videos there)
If the file needs to be downloaded (truly) from iCloud, then emit the progress in the PMProgressHandler as it was done in 1.3.10 please

Flutter version
[✓] Flutter (Channel stable, 2.8.1, on macOS 12.1 21C52 darwin-x64, locale fr-FR)
    • Flutter version 2.8.1 at /Users/foxtom/Desktop/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 77d935af4d (3 weeks ago), 2021-12-16 08:37:33 -0800
    • Engine revision 890a5fca2e
    • Dart version 2.15.1

[✓] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    • Android SDK at /Users/foxtom/Library/Android/sdk
    • Platform android-31, build-tools 31.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 13.2.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.11.2

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] VS Code (version 1.63.2)
    • VS Code at /Users/foxtom/Desktop/Visual Studio Code.app/Contents
    • Flutter extension version 3.32.0

[✓] Connected device (3 available)
    • Now You See Me (mobile) • 00008020-001204401E78002E • ios            • iOS 15.2 19C56
    • macOS (desktop)         • macos                     • darwin-x64     • macOS 12.1 21C52 darwin-x64
    • Chrome (web)            • chrome                    • web-javascript • Google Chrome 97.0.4692.71

• No issues found!

Smartphone (please complete the following information):

Device: iPhone XS Max, iPhone 12, iPhone 13 pro max
OS: iOS 14+ - iOS 15+

First problem : assetEntity.file or any other method is sometimes fetching the file but i have iCloud disabled on my iPhone and never put any files there, tested on 3 different iPhones without iCloud and all have the download but not all files.

This is out of support. We cannot control how a file behaves or where it's been stored.

Can you use git bisect to check commits between 1.3.10 and 2.0.0? I don't have much time since we're at the end of the lunar year and have massive work to deal with.
More specifically, git bisect start 947cf50 1dad659.

Same using image_picker actually it doesn't download.

That's why we have AssetEntity, and once you start to retrieve the file, it should be fetched.

I understand, but when i use  XFile? file = await picker.pickVideo(source: ImageSource.gallery); and then click on the play button, it's instantly playing the video and is not fetched.
The same video picked above is fetched during few seconds using assetEntity.file once the first time.
But after the first time it's always fluent and working very fine yes.
I simply don't want my iOS users to see for the first time in their life downloading asset... in my UI when the same file is available directly in other apps...  Especially if they don't use iCloud.
I will try to find some time to take a look at the code and understand better the Swift API, i am not saying that there is a bug here but since the behavior is different in other apps it simply makes me wonder if there is another way to do so. 90% of my iOS beta testers complained about it :/

Please file separate issues to track different things.

I didn't reproduce this issue either with iCloud or non-iCloud files.
Files with iCloud:
2022-01-17 13:11:15.271918+0800 Runner[17598:6803905] PhotoManager info: cache path = /private/var/mobile/Containers/Data/Application/83E1B5F6-2133-4E16-8895-453A27C3A3F7/tmp/.image
2022-01-17 13:11:15.280449+0800 Runner[17598:6803079] flutter: progress state onChange: PMRequestState.prepare, progress: 0.0
2022-01-17 13:11:16.082437+0800 Runner[17598:6803079] flutter: progress state onChange: PMRequestState.loading, progress: 0.97
2022-01-17 13:11:16.104184+0800 Runner[17598:6803079] flutter: progress state onChange: PMRequestState.loading, progress: 0.99
2022-01-17 13:11:16.134308+0800 Runner[17598:6803079] flutter: file = File: '/private/var/mobile/Containers/Data/Application/83E1B5F6-2133-4E16-8895-453A27C3A3F7/tmp/.image/(null)'
2022-01-17 13:11:16.134564+0800 Runner[17598:6803079] flutter: progress state onChange: PMRequestState.success, progress: 1.0

Files that already locally available:
2022-01-17 13:12:00.283762+0800 Runner[17598:6804273] PhotoManager info: cache path = /private/var/mobile/Containers/Data/Application/83E1B5F6-2133-4E16-8895-453A27C3A3F7/tmp/.image
2022-01-17 13:12:00.284935+0800 Runner[17598:6803079] flutter: progress state onChange: PMRequestState.prepare, progress: 0.0
2022-01-17 13:12:00.289065+0800 Runner[17598:6803079] flutter: progress state onChange: PMRequestState.success, progress: 1.0
2022-01-17 13:12:00.289318+0800 Runner[17598:6803079] flutter: file = File: '/private/var/mobile/Containers/Data/Application/83E1B5F6-2133-4E16-8895-453A27C3A3F7/tmp/.image/IMG_0215.JPG'

Thanks a lot for your time.
I saw that you are using getMediaUrl on the AssetEntity, and it gives the expected UI behavior, instantly played when not from iCloud and with edited changes ! :).
This actually solve #678 in UI perspective.
However, it's doesn't seem possible to upload it (my goal is to upload it like on any app afterwards) because this mediaUrl is not a File / bytes (File(mediaUrl).existsSync() returns false).
If i get the assetEntity.file right before upload (and use getMediaUrl for the preview only), i will not upload the edited file but the original one, even if i use loadFile(isOrigin: false).
Do you know how can we actually use this mediaUrl to upload the file please ?

I'm not sure how you combine these issues together. In #678 you're indicating an editing issue rather than the loading one.
getMediaUrl returns the ContentURI and the file URL of AVAsset which is playable from the native, as documented:
https://github.com/fluttercandies/flutter_photo_manager/blob/48dd4af7e100463c2ecbc09d3302c738662fd6be/lib/src/types/entity.dart#L593-L597
It's not related to whether the resource is ready for I/O usage.
If you don't have further issues with PMProgressHandler, please close this issue. Otherwise, let's focus on one perspective at a time.

Still unable to reproduce the issue. Closing as it's not reproducible.

iCloud file
non-iCloud file

Fixed by https://github.com/fluttercandies/flutter_photo_manager/pull/1174


Helpers not working

Devise 1.4.8, Rails 3.1, Ruby 1.9.2
Just installed and none of the helper methods are working? Any idea what this might be?
ActionView::Template::Error (undefined method `user_signed_in?' for #<#<Class:0xb15a4e0>:0xb15795c>)

Make sure you run the install generator, and configured your routes and model correctly. Please use the mailing list for questions.


plnkr for using custom loader service is broken

The plnkr for using custom loader service is throwing errors.
Uncaught TypeError: $translateProvider.uses is not a function from myApp

$translateProvider.uses -> $translateProvider.use, but already since v2.
@fahussain  Which plnkr you have used? Where did you find it?

Closed due inactivity.


Indexation

Hi,
To find easily your project, could you add these topics:
silverbullet
silverbullet-plugs
plantuml
....
Morevover you can add your plugins in https://github.com/gorootde/silverbullet-collection
Thanks for your work

You are right.

Updated topics.
@juandjara was faster and added your mindmap plugin and this plugin to https://github.com/gorootde/silverbullet-collection


Improve documentation for autoDispose

Describe what scenario you think is uncovered by the existing examples/articles
The autoDispose section of the article says the following:

To tell Riverpod to destroy the state of a provider when it is no-longer used, simply append .autoDispose to your provider:

I was troubleshooting an app bug related to auto-dispose. When a user was logging out, some of the state for the logged in user wasn't getting destroyed.
It turned out that another provider was depending on it using ref.watch which was preventing the state from being destroyed.
It was unclear to me what "no-longer used means". I think clarifying how riverpod decides what is still in use could help. Maybe with some examples of gotchas.
Describe why existing examples/articles do not cover this case
I think the examples do cover autoDispose, but because there are some gotchas and details on provider lifecycle to be aware of.
Additional context
I would like to confirm the following is the case:

Something is used if it's being referenced by a widget with "useProvider" OR another provider is in use with read or watch
If something is disposable, dispose will be called on it. (Would be good to in understand what cases river_pod will dispose something for you without manually calling dispose
ref.onDispose is additional cleanup to whatever river_pod does. (It doesn't override river_pod dispose)

The troubleshooting steps I wish I had known when autoDispose isn't triggering onDispose

Check that the provider isn't being referenced in any widgets
Check that other active providers aren't referencing it

Another idea would be to list all dependencies (widgets+providers) that a provider currently depends on in the DevTools inspector so we can easily see why something is still alive when it should have been disposed.
I can help out with documenting this once my understanding of autoDispose improves.

Agreed
I don't think documentation can help much here. What we are looking is for devtools instead.
A devtool is planned and is my next step after making a few more examples.
It would include this scenario.
But in the short term, the 0.7.0 version will have some improvements about common mistakes around ref.watch
You can see the changelog/doc for this

Ok awesome! Feel free to mark this as closed then.
The main use cases for me are:

What depends on provider A? (useful for knowing what is preventing autoDispose from being called, also includes Widgets in the widget tree)
What does A depend on? (useful for knowing what will rebuild if A changes)

I will keep this open as the devtool isn't technically here yet and the change made doesn't prevent all mistakes.

Related to this in the meantime would be useful to have a helper like
dependsOn(provider) and dependenciesOf  (don't know the exact naming) so I can print them out while debugging. Does something like this exist in the API? One of my most common debugging scenarios is finding out why something isn't autoDisposed.

You can do that. It's just not obvious
final provider = Provider(...);

ProviderContainer container = ProviderScope.containerOf(context);

container.debug*

final element = container.readProviderElement(provider);
print(element.dependents);

Agreed!!  I love Riverpod and one improvement I'd like to see in the documentation is clarity on HOW LONG a provider will live, WITHOUT an active listener, before it will self-dispose / garbage-collect.
I'm probably inventing some bad practices because I don't understand this fully ...
https://stackoverflow.com/questions/64830479/flutter-riverpod-design-pattern-inhibit-garbage-collection

I'd like to see in the documentation is clarity on HOW LONG a provider will live, WITHOUT an active listener, before it will self-dispose / garbage-collect.

The provider will be disposed at the end of the current frame/event-loop.
Riverpod does something roughly equivalent to:
@override
void removeListener(listener) {
  _listener.remove(listener);
  if (!hasListeners) {
    Future(() {
      if (!hasListeners) {
       dispose();
      }
    });
  }
}

This is pseudo code, but should help understand what happens.

I'd like to see in the documentation is clarity on HOW LONG a provider will live, WITHOUT an active listener, before it will self-dispose / garbage-collect.

The provider will be disposed at the end of the current frame/event-loop.
Riverpod does something roughly equivalent to:
@override
void removeListener(listener) {
  _listener.remove(listener);
  if (!hasListeners) {
    Future(() {
      if (!hasListeners) {
       dispose();
      }
    });
  }
}

This is pseudo code, but should help understand what happens.

That's very helpful ... thank you!!
a few clarifications please ...
Does this apply to ALL providers or ONLY those defined auto-dispose?
"watch" attaches a listener but "read" does not ... is that correct??
So if I force a Provider to initialize with context.read(), but DO NOT watch it, then "removeListener" will never be called and this (non-autodispose) provider will live forever??

That's very helpful ... thank you!!
a few clarifications please ...
Does this apply to ALL providers or ONLY those defined auto-dispose?
"watch" attaches a listener but "read" does not ... is that correct??
So if I force a Provider to initialize with context.read(), but DO NOT watch it, then "removeListener" will never be called and this (non-autodispose) provider will live forever??

If you call "read" and it no listener is attached, the state will still be disposed at the end of the frame
And that's only for autoDispose providers

If you call "read" and it no listener is attached, the state will still be disposed at the end of the frame
And that's only for autoDispose providers

Following up on this thread. A very common source of debugging is understanding why a provider isn't disposed. So I'd love to see what values depend on a provider--what widgets and other providers are currently making use of this provider?

This is planned for the devtool.
On top of inspecting the state, you will be able to inspect the links between objects. So you'd know what is listening to a provider

That would be amazing :)
Happy to be a beta tester for the devtool, but we're usually on the bleeding edge of riverpod anyway :)

Not sure of relevant for this topic, but calling .dispose() does not dispose a provider, correc? I can't find any other reference on how to manually dispose a non autodipose provider. Maybe my approach to the design is wrong and I'm overcomplicating it.

It was unclear to me what "no-longer used means".

I totally agree with this, came here from google. As a developer coming to riverpod I need guarantees about when the provider will be destroyed, and when not. For two reasons: To make clear predictions about my applications behavior and to have assurance that .autoDispose is reliable.
As great as tooling is, documenting the guarantees and behavior .autoDispose should come first. It's hard to reason about your app when you are basing your expectations on assumptions.

This PR https://github.com/rrousselGit/riverpod/pull/1689 is intended to address some of the documentation requests in this issue. If you want to leave feedback there, that would be helpful to understand whether there is more that needs to be explained.

#1689 was merged, so this probably can be closed.


feat: add code to errors

This feature will allow us to support translations for error messages.

The code property of an error should be a short string that could be used to lookup a translation for what error message should be shown to a user.
Error codes are normalized to be UPPER_CASE+SNAKE_CASE so we can have some consistency. For example, ERR_RATE_LIMIT.
All errors have a default code that matches the error's name.

I like it.
I'd prefer to see ERROR instead of ERR in all instances, but that's just stylistic.

I was just following how node.js does their errors.


Move to Gradle 6.8.1 and use java toolchains

Updates the Gradle wrapper to 6.8.1 and moves the project to use the new Java Toolchains feature.
In short, Java Toolchains, added in Gradle 6.7 and enhanced in 6.8, is a feature that allows Gradle projects to define what JDK versions to use for the project and/or individual tasks, and automatically downloads them if missing and uses those JDKs.
This allows for developers for ForgeFlower to not have to install JDK 8 and/or 9 by themselves, by having Gradle do the downloading automatically. (The installed JDKs and the downloaded JDKs are used in other projects that use Java Toolchains too, when needed.)
This also includes a small fix to a bug I've been encountering where deleting the built JARs or even the build folder in the ForgeFlower subdirectory does not allow recompilation of the classes and the rebuilding of the JARs, by forcing the jar task (and any other Jar task such as sourcesJar) to always run and ignore the UP-TO-DATE. This shouldn't really affect anything, as it only goes into effect when you build the project. See comment below.
Also has some general buildscript cleanup.

Cleans up the formatting in places (trailing whitespace).
Updates Grgit to 4.1.0, and adds tags: true to the grgit.describe call.
Changes how the eclipse task is created/edited, because Eclipse apprently applies the eclipse plugin automatically; it now creates the task only if it does not exist first.

Per feedback from Lex, I've removed the caching issue bugfix/workaround described in the second-to-last paragraph of the PR description, as it still needs further investigation as to the actual root cause of the problem.

Upon further investigation (using a seperate Linux install), I've determined the issue to be local to my Windows installation on my machine. Therefore, the fix/workaround will remain removed.

Sorry I forgot about this, and thanks for doing it. Just a cleanup I never got time to do.


Main Page & Basic Layout

Value Proposition
As a user, I want to see a main page when I have access to the app so that I can see the content which the app provides.
Description

##Acception Criteria

[ ] I want to have a header-element in the app (no functionality yet)
[ ] I want to have a footer-element in the app (no functionality yet)
[ ] I want to see an overview of the content/categories the app has to offer

Tasks

[ ] Create initial page
[ ] Add dummy header
[ ] Add dummy footer
[ ] Add dummy welcome headline and intro text
[ ] Add container components for categories
[ ] Add dummy headline to the containers
[ ] Add dummy text to the containers
[ ] Add basic CSS

In your wireframe is a lot more to see than to do in the tasks. I see a burger-menu, a login welcome, a footer...

Roland pointed out the main issue with this user story, it is more like an epic, I guess you won't see this stage of your app for quite a quile. Remember: everything that is displayed in your app needs to be functional  and working. If you don't have an extra page where lessons are displayed, don't add a link to that page. Same for game plan and statistics.
I guess you have to start at another part of your App. Don't imagine what it will be like and code the first part the user might see when they open the app. Start with the most fundamental feature of your App, a list of lessons or a display of a game plan for example. Your App will "grow" naturally form that point on.


Module name

In the doc, you say "use fv.tipped" module but the correct name seems to be "decipher.tipped"
EDIT : on Github, the module name is "fv.tipped", my bad, maybe a branch problem ?
EDIT 2 : Ok, I don't know why bower gives me the v0.3.0 version... Updated to v1.0.0-1.

@thco v1.0.0-1 is an release candidate for 1.0.0 until we can update the documentation with new features and behavior.  Feel free to use it though, and let me know if you find issues.  see #17


Fix different positional embeddings clashing

As mentioned on discord, if using rpe, the regular pos embs were still being added in.
This should fix that - you now specify the type of positional embedding you want as the "--pos-emb" arg and can choose from 'learned', 'sinusoidal' and 'rpe'.
Still needs testing.

We might want to double check our sinusoidal implementation, but all the others apppear to work fine.

dashed peach line:  sinusoidal,
other peach line : learned embedding summed w input,
green : no pos emb
grey : rpe

Are we going to leave this open until the bug in sinusoidal is fixed or would you rather we merged and opened a bug issue @sdtblck

Are we going to leave this open until the bug in sinusoidal is fixed or would you rather we merged and opened a bug issue @sdtblck

It's unclear whether it's a bug but if it is, it's in the main branch too. And the rpe stuff just flat out doesn't work in the main branch, so i'd suggest we merge this.


Cleanup

Codecov Report

Merging #27 into master will decrease coverage by -0.73%.
The diff coverage is 83.96%.

@@            Coverage Diff             @@
##           master      #27      +/-   ##
==========================================
- Coverage   93.36%   92.63%   -0.73%
==========================================
  Files          11       12       +1
  Lines        1748     1807      +59
==========================================
+ Hits         1632     1674      +42
- Misses        116      133      +17

Impacted Files
Coverage Δ

Sources/PostgreSQL/Node+Binding.swift
83.15% <ø> (ø)
:white_check_mark:

Sources/PostgreSQL/Node+Oid.swift
98.2% <100%> (ø)
:white_check_mark:

Sources/PostgreSQL/Database.swift
81.81% <100%> (-8.81%)
:x:

Sources/PostgreSQL/Result.swift
100% <100%> (ø)
:white_check_mark:

Tests/PostgreSQLTests/Utilities.swift
64.28% <100%> (ø)
:white_check_mark:

Tests/PostgreSQLTests/PostgreSQLTests.swift
95.5% <100%> (ø)
:white_check_mark:

Sources/PostgreSQL/Status.swift
25% <100%> (ø)
:white_check_mark:

Sources/PostgreSQL/ConnectionInfo.swift
33.33% <33.33%> (ø)

Sources/PostgreSQL/Connection.swift
79% <78.2%> (-4.34%)
:x:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9c0c3a...cd6b142. Read the comment docs.


Creating an empty table in mysql

Current behavior
While working on #1594, I noticed that creating a table without any column (unusual, isn't it? 😉 ),
the generated SQL query for MySQL is invalid:
  defmodule DuntMigration do
    use Ecto.Migration

    def change, do
      create table(:dummy)
    end
  end

=> CREATE TABLE "dummy" () ENGINE = INNODB;

Expected behavior
It should generate CREATE TABLE "dummy" ENGINE = INNODB;.
I'll gladly do it, feel free to assign me.

@xcambar i can't  assign you to it but a PR is welcome :D


Blender 2.82a glTF 1.2.18 Exports PSD as JPG, there is no PNG option any more

Blender 2.82a with Manually updated glTF Exporter 1.2.18 from Blender 2.83 (1.2.18) as well as the downloaded 1.2.48 from git repo.
Need the exported textures to be in PNG format from my working PSD textures, can't have them being saved to JPG as we need to do arbitrary post export compression of textures to work within our games size limits.
Bug Blender 2.82a glTF 1.2.18 Exports PSD as JPG, there is no PNG option any more.zip

This was changed by #882.
It seems that though Blender can open PSDs, internally it just converts it to a JPEG, which is why the exporter treats it like one. If you check the image.file_format field it's 'JPEG', if you go to "Save As" Blender will try to save it as a JPEG, etc.
What we could do is check if the image came from a file and that file ends in .psd/.PSD`, and use PNG in that case.
This also means it's possible to have an image that's JPEG format, but whose file on disk isn't a JPEG. Will need to check that that's handled correctly...


Agentless scanning can be enabled with subscription Contributor

According to this page, Agentless scanning can be enabled with subscription owner, but I confirmed it is possible with subscription Contributor as well. Is the following description correct?

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 2adc522b-6260-65dd-dff6-8e08bf951541
Version Independent ID: 63ea0808-37c4-f7cf-8721-3518ab8eb0f7
Content: Overview of Cloud Security Posture Management (CSPM)
Content Source: articles/defender-for-cloud/concept-cloud-security-posture-management.md
Service: defender-for-cloud
GitHub Login: @ElazarK
Microsoft Alias: elkrieger

@Yuki-Nishio thank you for bringing this to our attention. I will look into this and hopefully will have an answer for you soon.

@Yuki-Nishio
Thanks for bringing this to our attention.
I'm going to assign this to the document author so they can take a look at it accordingly.

@AjayBathini-MSFT
I'm sorry, I misunderstood it. I confirmed a contributor cannot enable agentless scanning. I thought I was able to enable it because successful message was outputted when doing it. Could you abandon this request?

@Yuki-Nishio
if there are any further questions regarding the documentation, please tag me in your reply and we will be happy to continue the conversation.


add direct link to claim page

Adds a direct link to claim page in addition to the QR.

#6 will be closed after deployment. Closing this PR.


Fix no_std build failure

{f32|f64}::round() does not exist with just core, so libm's implementation is used instead when the std feature is not activated.

I have no idea what's going on here tbh. cargo test --no-default-features with
#[cfg(not(test))]
use libm::{F32Ext, F64Ext};

spits out
warning: unused import: `F32Ext`
   --> palette/src/lib.rs:172:12
    |
172 | use libm::{F32Ext, F64Ext};
    |            ^^^^^^
    |
    = note: #[warn(unused_imports)] on by default

warning: unused import: `F64Ext`
   --> palette/src/lib.rs:172:20
    |
172 | use libm::{F32Ext, F64Ext};
    |                    ^^^^^^

Hmm... strange, but good catch! I wonder how it passed the tests in the first place. Did you compile with any specific feature set, or just all of them disabled?
Either way, the float trait business is a bit weird and convoluted, but take a look in the float.rs file. That's where we have all the mappings to libm, and that's where you can add this code too.

Because the tests are always built with std.  I forgot about that. :man_facepalming: Classic and not so strange after all. Do you think you can add a compile tests in the test_features.sh file? Both check this and future mistakes.

Unfortunately this gets even harder because the lock file also contains the dev dependencies which make the command I posted also not work due to the cargo bug where features of dev dependencies leak into your normal dependencies. So we would need a separate crate specifically to test no_std compatibility that depends on palette. The state of no_std in Rust is honestly still pretty sad tbh.

That's annoying. One would think the dev dependencies would be isolated. In that case it may be better if it's added in a separate issue and PR, to keep the scope of this one focused. I have obviously not investigated the general no_std situation enough, but I suppose it would be good to try to test for it not being leaked, too. So it's doesn't sound like a quick addition.

Too late, that's already gone :P

You are too fast for me 😁

The builds look good so far, so I'll approve this. I'll also create an issue for setting up no_std tests.

bors r+


Assertion failure in SafepointHandler::ExitSafepointUsingLock

https://build.chromium.org/p/client.dart/builders/vm-win-debug-ia32-be/builds/1592/steps/vm tests/logs/stdio
https://build.chromium.org/p/client.dart/builders/vm-win-debug-ia32-be/builds/1593/steps/vm tests/logs/stdio
Loading VM...
Done loading VM
Running observatory_assets_test.dart [1/1]
Trying to fetch http://localhost:62216/third_party/trace_viewer_full.html
vm\safepoint.cc: 173: error: expected: T->IsAtSafepoint()
** Killing script

@a-siva

Duplicate of #27006.


Document new dependency management features

WIP documentation here: https://docs.google.com/document/d/1Cfh3CeepC3cZuRBCXVnxfR1vyCu7wFmFBYneXedcaIE/edit#heading=h.uo6da3819lzn
Features include:

strict dependencies
dependency constraints
new module metadata
BOM import
Optional dependencies
Configuration.withDependencies

Once ready, cover in:

Release notes
User guide

The document linked above has been deleted and the content was moved.
For 4.6 we covered:

dependency constraints
BOM import
Optional dependencies

Follow up issues for remaining topics:

strict dependencies: part of #3977
new module metadata: #4049
Configuration.withDependencies: #4377


.format("cosmos.oltp") conflict between azure-cosmos-analytics-spark_3-4_2-12_synapse-2.1.1.jar and azure-cosmos-spark_3-4_2-12-4.34.0.jar

I wish use use the latest version of spark cosmos connector, but it appears the default spark pool configuration  in Azure Synapse already has a package that provides .format("cosmos.oltp")  by default.
As far as I can establish that package is azure-cosmos-analytics-spark_3-4_2-12_synapse-2.1.1.jar
I want to use latest and greatest azure-cosmos-spark_3-4_2-12-4.34.0.jar
I am unable to configure spark session with the default package removed. spark.jars.excludes won't remove it, probably because it's a default.
With the newer package added, how can I be certain that it is the one being used when I run the cosmos write job ?
Background:
We're using ThroughputControl with targetThroughput but only achieving about 25% of the target, though we can get much higher throughput on that container without ThroughputControl.
Using the very latest spark connector feels like the right thing to pursue given how far out of date the default one is.

@golfalot can you please share the config in your Scala/Python notebook here (preferably the full code if feasible), and the throughput in your container? We'll address the spark connector versioning issue separately (I doubt it has a bearing on your issue with throughput control).

Thank you @TheovanKraay
We're seeking a solution where we can load fast but without saturating RUs to avoid the clients apps getting any 429 during the bulk load/refresh.
We are writing 1m Structs (scaled down for test)  to a single partition (I know... but needs must as data api builder doesn't currently support hierarchical partitions).
The Cosmos DB is set to autoscale max 31k RUs shared. Single container means we're capping out at a theoretical 10k max RUs. We understand that limitation.
# Query cores per executor
cores_per_executor = sc.getConf().get("spark.executor.cores")
print(f"Cores per executor: {cores_per_executor}")

num_executors = int(spark.conf.get("spark.executor.instances"))
print(f"Number of executors: {num_executors}")

# Calculate total number of partitions
total_cores = int(num_executors) * int(cores_per_executor)

path = "/gold/price_paid/factors_with_category_type_with_history_cosmos"

# read in data
df_1m_sample = (
    spark.read.format('delta')
    .load(path).drop('version')
    .withColumn('version', F.lit('data_202408_abp_107_build_v1'))
    .filter(F.col('price_paid_version') == '202408')
    .withColumnRenamed('price_paid_version', 'data_version')
    .withColumn('id', F.col('uprn').cast('string'))
    .orderBy(F.col('uprn'))
    .limit(1000000)
    .repartition(total_cores)
)

# force into memory for testing only
df_to_write = spark.createDataFrame(df_1m_sample.collect(), schema=df_1m_sample.schema)

This will saturate at Max Autoscaled RUs at 7.75k RUs, but with many 429 errors
(
    df_to_write
    .write
    .format("cosmos.oltp")
    .option("spark.synapse.linkedService", "CosmosDB_GraphQL_rating_datasets_by_source_01_to_25")
    .option("spark.cosmos.container", 'pricePaidFixedRUv2')
    .mode("APPEND")
    .save()
)

This only reaches 2.26 RUs
(
    df_to_write
    .write
    .mode("APPEND") # do not delete container before writing
    .format("cosmos.oltp")
    .option("spark.synapse.linkedService", "CosmosDB_GraphQL_rating_datasets_by_source_01_to_25")
    .option("spark.cosmos.container", 'pricePaidFixedRUv2')
    .option('spark.cosmos.throughputControl.enabled','true')
    .option("spark.cosmos.write.bulk.initialBatchSize", 1)
    .option('spark.cosmos.throughputControl.name','pricePaid_Loader_Jobs_Group')
    .option('spark.cosmos.throughputControl.globalControl.database','<redacted>')
    .option('spark.cosmos.throughputControl.globalControl.container','ThroughputControl')
    .option('spark.cosmos.throughputControl.accountKey',TokenLibrary.getSecretWithLS("<redacted>","read-write-cosmos-elastic-reference-data-uks")) # need to use a proper secret here, but variable ok for
    .option('spark.cosmos.throughputControl.targetThroughput', 9500)
    .save()
)

throughput container snapshot
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtOTUwMA91ad6c1e-92a9-4db9-92ec-b940a7ee6f9a",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"0b06f411-0000-1100-0000-673601b60000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T13:29:25.754977Z",
        "loadFactor": 1.1566333920010767,
        "allocatedThroughput": 4634.486770433088,
        "_rid": "fawUALp-HLmfAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmfAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtOTUwMA0575a16f-4e17-4c5a-8d4e-e7b99e46077d",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"0b06d211-0000-1100-0000-673601b60000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T13:29:25.649294Z",
        "loadFactor": 1.2142908911603594,
        "allocatedThroughput": 4998.574744514014,
        "_rid": "fawUALp-HLmeAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmeAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3R0LTAuMw.info",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.config",
        "targetThroughput": "",
        "targetThroughputThreshold": "0.3",
        "isDefault": false,
        "_rid": "fawUALp-HLmKAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmKAAAAAAAAAA==/",
        "_etag": "\"b1058a1b-0000-1100-0000-6734dacf0000\"",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    }

doing some "dumb" .option('spark.cosmos.throughputControl.targetThroughput', 28000)  and unintuitive gives us 6.12k RUs sustained
here are the clients
[
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtOTUwMA0575a16f-4e17-4c5a-8d4e-e7b99e46077d",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"0e06ddc3-0000-1100-0000-673606cd0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T13:29:25.649294Z",
        "loadFactor": 0.1,
        "allocatedThroughput":<PHONE_NUMBER>6277,
        "_rid": "fawUALp-HLmeAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmeAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtOTUwMA91ad6c1e-92a9-4db9-92ec-b940a7ee6f9a",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"0e06d9c3-0000-1100-0000-673606cd0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T13:29:25.754977Z",
        "loadFactor": 0.1,
        "allocatedThroughput":<PHONE_NUMBER>6277,
        "_rid": "fawUALp-HLmfAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmfAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtMjgwMDA0fed1b45-bfd5-4522-b8b4-36fd48f8ba7e",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"0e0619c5-0000-1100-0000-673606cf0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T14:02:53.035802Z",
        "loadFactor": 1.146207023576095,
        "allocatedThroughput": 12285.190558949238,
        "_rid": "fawUALp-HLmiAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmiAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtMjgwMDA03deee1b-89b2-4a28-95a1-9ec2de0eb988",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"0e0606c5-0000-1100-0000-673606cf0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T14:02:53.175663Z",
        "loadFactor": 1.2661901147614698,
        "allocatedThroughput": 13029.794957771293,
        "_rid": "fawUALp-HLmjAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmjAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    }
]

@golfalot thank you for sharing the information. First, can you try switching to dedicated container throughput instead of shared db throughput? Shared throughput does have some edge case issues. There should also be less need for this since min throughput can now be set to 1000RUs max for autoscale (min 100 RU per container). Just want to rule out issues with dedicated RU/confirm whether you are seeing the same results for dedicated RU before recommending/investigating anything further.

following on with the unintuitive,  .option('spark.cosmos.throughputControl.targetThroughput', 24000) (8% decrease) achieves an avg 5.5k RUs scale, no 429 errors
[
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtOTUwMA0575a16f-4e17-4c5a-8d4e-e7b99e46077d",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"1606ecd0-0000-1100-0000-673612be0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T13:29:25.649294Z",
        "loadFactor": 0.1,
        "allocatedThroughput":<PHONE_NUMBER>4830756,
        "_rid": "fawUALp-HLmeAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmeAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtOTUwMA91ad6c1e-92a9-4db9-92ec-b940a7ee6f9a",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"1606b6d0-0000-1100-0000-673612be0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T13:29:25.754977Z",
        "loadFactor": 0.1,
        "allocatedThroughput":<PHONE_NUMBER>2805505,
        "_rid": "fawUALp-HLmfAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmfAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtMjgwMDA0fed1b45-bfd5-4522-b8b4-36fd48f8ba7e",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"160684d1-0000-1100-0000-673612bf0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T14:02:53.035802Z",
        "loadFactor": 0.1,
        "allocatedThroughput": 1041.0057181213276,
        "_rid": "fawUALp-HLmiAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmiAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtMjgwMDA03deee1b-89b2-4a28-95a1-9ec2de0eb988",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"16068ed1-0000-1100-0000-673612bf0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T14:02:53.175663Z",
        "loadFactor": 0.1,
        "allocatedThroughput": 1041.0057181213276,
        "_rid": "fawUALp-HLmjAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmjAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtMjQwMDAfa7bd8d1-28c7-4dc5-868f-63e9f18aaf12",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"1606ded0-0000-1100-0000-673612be0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T14:43:02.051361Z",
        "loadFactor": 1.2365507777945268,
        "allocatedThroughput": 11033.626546555519,
        "_rid": "fawUALp-HLmmAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmmAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    },
    {
        "id": "cmF0aW5nLWRhdGFzZXRzLWJ5LXNvdXJjZS0wMS0yNS9wcmljZVBhaWRGaXhlZFJVdjIvcHJpY2VQYWlkX0xvYWRlcl9Kb2JzX0dyb3VwL3QtMjQwMDA53ea2b2e-48b6-4b0a-93ed-98724acf433d",
        "groupId": "rating-datasets-by-source-01-25/pricePaidFixedRUv2/pricePaid_Loader_Jobs_Group.client",
        "_etag": "\"1606ced0-0000-1100-0000-673612be0000\"",
        "ttl": 11,
        "initializeTime": "2024-11-14T14:43:02.116834Z",
        "loadFactor": 1.0531558695936143,
        "allocatedThroughput": 9274.669043653816,
        "_rid": "fawUALp-HLmnAAAAAAAAAA==",
        "_self": "dbs/fawUAA==/colls/fawUALp-HLk=/docs/fawUALp-HLmnAAAAAAAAAA==/",
        "_attachments": "attachments/",
        "_ts":<PHONE_NUMBER>
    }
]

Hi @TheovanKraay good idea, here are the latest findings
on db scoped provisioned autoscale, we increased from 31k to 40k, but still only achieved 7.5k on the container with .option('spark.cosmos.throughputControl.targetThroughput', 40000)
on per container autoscale 10k max, we can indeed now reach 10k autoscale. So that's a win! Only downside is there was a brief period where it overshot and cause 1.37k 429s during a one minute window before throttling back a little. .option('spark.cosmos.throughputControl.targetThroughput', 9500)
We're going to experiment with the targetThroughput value to try to prevent overshoot.
We've also noticed another interesting feature. When runnig all these tests in interactive notebook session in Synpase, the client documents/items from previous runs continue to exist. Not until the spark session is terminated/stopped does the ttl get honoured

@golfalot per the docs - throughput control doesn't do RU pre-calculation of each operation, as it is implemented client-side. Instead, it tracks the RU usages after the operation based on the response header. So its an approximation, and doesn't guarantee that amount of throughput is available for the group at any given time. The kind of throttling anomaly you described is more-or-less expected - to eliminate it, you would probably need to set a lower limit. The docs in the throughput control container are metadata only, and should be getting updated (and ttl purged) on each new run.

@TheovanKraay understood about the overshoot, understand the reasons.
ref: The docs in the throughput control container are metadata only, and should be getting updated (and ttl purged) on each new run.
Can confirm definitely not the case with an interactive session and multiple cosmos runs on the same spark session.
Thanks for your prompt responses today, we are in a much better place thanks to you!


INT B-20787 NTS/NTS-Release destination/pickup addresses are set to storage facility address

B-20787
B-20788
B-20789
Summary
Updates the mto_shipment service to apply the user inputted storage facility address to the pickup and destination addresses of NTS-Release and NTS shipments respectively.
This also completed B-20788 and B-20789 without any additional changes. Testing instructions are provided below to verify their AC has been met with this PR.
Verification Steps for the Author
These are to be checked by the author.

[ ] Tested in the Experimental environment (for changes to containers, app startup, or connection to data stores)
[ ] Have the Agility acceptance criteria been met for this change?

Verification Steps for Reviewers
These are to be checked by a reviewer.

[ ] Has the branch been pulled in and checked out?
[ ] Have the BL acceptance criteria been met for this change?
[ ] Was the CircleCI build successful?
[ ] Has the code been reviewed from a standards and best practices point of view?

Setup to Run the Code

Instructions for starting storybook
Instructions for starting the MilMove application
Instructions for running tests

How to test
B-20787

Access the customer application and create a new move
Add an NTS and NTS-Release shipment to your move taking note of the storage facility address you supplied for the NTS shipment
Log in a a Services Counselor and open the move you created
Optionally check the network tab to see if the storage facility address you supplied for the NTS shipment appears in the destination address of that shipment.
Fill out required fields and approve both shipments.
Log in as a TOO
Fill out the storage facility for the NTS-Release shipment and note the address.
Approve both moves as a TOO.
Check the network tab to see that both moves now have populated pickup, destination, and storage facility addresses and that the pickup and storage facility address match for the NTS-Release and the destination and storage facility match for the NTS.
Optionally pause here to verify the AC for 20788 (this can be done after finishing the main testing steps).
Log into the Prime Simulator.
Optionally pause here to verify the AC for 20789 (this can be done after finishing the main testing steps).
Attempt to update the pickup address for the NTS-Release shipment. You should not be able to. You should get an error something like "cannot update address directly, please update storage facility instead" unless a different restriction is prevents you first.
Attempt to update the destination address for the NTS shipment. You should not be able to. You should get an error something like "cannot update address directly, please update storage facility instead" unless a different restriction is prevents you first.
Optionally repeat for an unapproved NTS shipment, this one shouldn't have any other address update restrictions.

B-20788

Login as a TOO and open the move created above.
Navigate to the move task order page
Verify the storage facility you entered for the NTS shipment is displayed correctly in the addresses section for that shipment under both storage facility on the right and destination in the center.
Verify the storage facility you entered for the NTS-Release shipment is displayed correctly in the addresses section for that shipment under both storage facility on the right and pickup on the left.
Update the storage facility address for both shipments
Repeat steps 3-5 with the updated addresses

B-20789

Log into the Prime Simulator and open the move created above.
Verify the details of the NTS-Release shipment includes a pickup address (it was previously null) and that it matches the storage facility address you inputted above.
Click the button to edit addresses, verify the pickup address is populated and matches.
Verify the details of the NTS shipment includes a destination address (it was previously null) and that it matches the storage facility address you inputted above.
Click the button to edit addresses, verify the destination address is populated and matches.

Frontend

[ ] There are no aXe warnings for UI.
[ ] This works in Supported Browsers and their phone views (Chrome, Firefox, Edge).
[ ] There are no new console errors in the browser devtools.
[ ] There are no new console errors in the test output.
[ ] If this PR adds a new component to Storybook, it ensures the component is fully responsive, OR if it is intentionally not, a wrapping div using the officeApp class or custom min-width styling is used to hide any states the would not be visible to the user.
[ ] This change meets the standards for Section 508 compliance.

Backend

[ ] Code follows the guidelines for Logging.
[ ] The requirements listed in Querying the Database Safely have been satisfied.

Database
Any new migrations/schema changes:

[ ] Follows our guidelines for Zero-Downtime Deploys.
[ ] Have been communicated to #g-database.
[ ] Secure migrations have been tested following the instructions in our docs.

Screenshots

NTS shipment has storage address matching destination address, displays on MTO page, updates to addresses display.

Prime Sim NTS address shows, not able to update destination address,

"error": "destination address update requests can only be created for HHG and NTSr shipments"

Out of scope for this PR but noticed if you edit an NTS-Release shipment as a SC and enter the Storage address then click Add or edit codes button for Accounting codes, once you save or cancel the View orders page that loads you are taken back to the Move details page and the storage address you entered isn't saved. Its a bit annoying. I will see about creating an issue for this to addressed in a future backlog.

NTS-Release pickup address same as storage address. Address show on MTO page. Prime Sim shows address. Error shows when trying to edit pickup as Prime.

Out of scope for this PR but noticed if you edit an NTS-Release shipment as a SC and enter the Storage address then click Add or edit codes button for Accounting codes, once you save or cancel the View orders page that loads you are taken back to the Move details page and the storage address you entered isn't saved. Its a bit annoying. I will see about creating an issue for this to be addressed in a future backlog.

Noticed this as well. I think there are similar behaviors other places as well, where it's easy to lose a bunch of unsaved info.

Out of scope for this PR but noticed if you edit an NTS-Release shipment as a SC and enter the Storage address then click Add or edit codes button for Accounting codes, once you save or cancel the View orders page that loads you are taken back to the Move details page and the storage address you entered isn't saved. Its a bit annoying. I will see about creating an issue for this to be addressed in a future backlog.

Noticed this as well. I think there are similar behaviors other places as well, where it's easy to lose a bunch of unsaved info.

I think some of these can be resolved by modifying routing. I also noticed if you do the add codes I mentioned before and enter in a SAC and save it will take you back to the Move Details page and say that you need to add SAC even though you already did. You have to edit the shipment and click save in order for it to go away.

Looks good. Thanks for fixing the error message.


Proxy_pass issues

Hi Nishen,
Congratulations on the project.
I would like to apply the ad authentication in a host that is already been under a reverse proxy. My configuration is this:
upstream production {
ip_hash;
server <IP_ADDRESS>;
}
location / {
proxy_cookie_flags ~ secure samesite=strict;
proxy_set_header Host 'production.company.local';
proxy_pass http://production;
}
Is there a way to insert the authentication in this context? Thank you

Hi there,
If I understand this correctly, should absoloutely be able to do that. In fact the standard model for it working is exactly this. The example configuration gives you this scenario as well.

Yes, you right. I didn't notice that. Thank you


[SPARK-32318][SQL][TESTS] Add a test case to EliminateSortsSuite for ORDER BY in DISTRIBUTE BY

What changes were proposed in this pull request?
This PR aims to add a test case to EliminateSortsSuite to protect a valid use case which is using ORDER BY in DISTRIBUTE BY statement.
Why are the changes needed?
scala> scala.util.Random.shuffle((1 to 100000).map(x => (x % 2, x))).toDF("a", "b").repartition(2).createOrReplaceTempView("t")

scala> sql("select * from (select * from t order by b) distribute by a").write.orc("/tmp/master")

$ ls -al /tmp/master/
total 56
drwxr-xr-x  10 dongjoon  wheel  320 Jul 14 22:12 ./
drwxrwxrwt  15 root      wheel  480 Jul 14 22:12 ../
-rw-r--r--   1 dongjoon  wheel    8 Jul 14 22:12 ._SUCCESS.crc
-rw-r--r--   1 dongjoon  wheel   12 Jul 14 22:12 .part-00000-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel   16 Jul 14 22:12 .part-00043-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel   16 Jul 14 22:12 .part-00191-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    0 Jul 14 22:12 _SUCCESS
-rw-r--r--   1 dongjoon  wheel  119 Jul 14 22:12 part-00000-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  932 Jul 14 22:12 part-00043-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  939 Jul 14 22:12 part-00191-2cd3a50e-eded-49a4-b7cf-94e3f090b8c1-c000.snappy.orc

If we remove the inner ORDER BY, the file size increases.
scala> scala.util.Random.shuffle((1 to 100000).map(x => (x % 2, x))).toDF("a", "b").repartition(2).createOrReplaceTempView("t")

scala> sql("select * from (select * from t order by b) distribute by a").write.orc("/tmp/SPARK-32276")

$ ls -al /tmp/SPARK-32276/
total 632
drwxr-xr-x  10 dongjoon  wheel     320 Jul 14 22:08 ./
drwxrwxrwt  14 root      wheel     448 Jul 14 22:08 ../
-rw-r--r--   1 dongjoon  wheel       8 Jul 14 22:08 ._SUCCESS.crc
-rw-r--r--   1 dongjoon  wheel      12 Jul 14 22:08 .part-00000-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    1188 Jul 14 22:08 .part-00043-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel    1188 Jul 14 22:08 .part-00191-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc.crc
-rw-r--r--   1 dongjoon  wheel       0 Jul 14 22:08 _SUCCESS
-rw-r--r--   1 dongjoon  wheel     119 Jul 14 22:08 part-00000-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  150735 Jul 14 22:08 part-00043-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc
-rw-r--r--   1 dongjoon  wheel  150741 Jul 14 22:08 part-00191-ba5049f9-b835-49b7-9fdb-bdd11b9891cb-c000.snappy.orc

Does this PR introduce any user-facing change?
No. This only improves the test coverage.
How was this patch tested?
Pass the GitHub Action or Jenkins.

Could you review this, @viirya ?

Also, cc @cloud-fan and @HyukjinKwon .

Thank you, @maropu and @viirya .
Yes. The commit log and JIRA will explain the situation. I made the test case minimally.

Do you think it is easy to add a test that checks file size like in the description? Or current one is enough?

Actually, the file size check test cases are very flaky. We hit many issues before when we add Spark Version metadata on Parquet/ORC/Avro.

Do you think it is easy to add a test that checks file size like in the description? Or current one is enough?

okay, sounds good.

@dongjoon-hyun Tiny nit: The 2nd example still has the inner ORDER BY clause. We meant to remove it , right ?

Thanks, @dilipbiswal . That's also right. Initially, it was intentional. To be more clear, I will add a comment to refer SPARK-32318. Technically, if the user writes a code sql("select * from t distribute by a").write.orc("/tmp/SPARK-32276") without inner order by, Spark should generate a large file.

Thank you all. GitHub Action catalyst module passed. I'll merge this.


Fix compiler warnings

signed vs unsigned comparison
unused variable (variables were only used for asserts, and I was
compiling a release version)
static should be prior to const modifier

All of these are upstream libraries, so the PRs should be submitted against them. Timelib primarily lives in PHP, so that's the best place to submit that PR. The repos for the other two should be fairly easy to find.

libmbfl patch integrated on upstream:
https://github.com/moriyoshi/libmbfl/commit/5b426829a93cc4e75b0c223cda548dba6ca124cd


Juju status blocked

Hello,
After letting the maas cluster running for a day, i tried to do juju status on one node, but got message that I need to enter password for the host on anvill-controller.

@aym-frikha I think you included wrong screenshot. Anyhow, password is located in ~/snap/maas-anvil/current/account.yaml

Is it possible please to explain to me the benefit of this process where having the random password located in ~/snap/maas-anvil/current/account.yaml ? From user experience perspective I don't think it's ideal. specially if it's not documented

Hi @aym-frikha , With #34 we added the juju-login command that sunbeam is also using to manage this situation. We also added documentation notes on why and when this command is needed. Feel free to try the command during one of your next Anvil deployments. Thank you for detecting the issue in the first place :beer:


Add Kotlin version table

@bot-gradle test and merge

Your PR is queued. See the queue page for details.

OK, I've already triggered a build for you.


Filter images in S3 bucket

Hello,
I've a S3 bucket with a lot of images with random hashes as file names in a flat structure:
0a1cce8ea0954bd7a76e4f254aa9e39b
0a4ce4b98976408a9d54a17ed3053cc2
0a8fda330de5436295ce293f2b75fc76
...

I was successfully able to add the S3 bucket to LS & to sync all images inside the bucket.
However I'd need to filter out some images & have only a subset in LS - is there a way to define e.g. a list with files to import (regex will not work, because of the random hashes & flat structure).

Hello, you have two options:

Filter tasks in the Data Manager using task url fields (you will have lots of OR filters)
Don't sync your bucket, just add it and them import JSON file with tasks you need, like:

[
 { "url": "s3://bucket/0a1cce8ea0954bd7a76e4f254aa9e39b.jpg"},
 { "url": "s3://bucket/0a4ce4b98976408a9d54a17ed3053cc2.jpg"},
 { "url": "s3://bucket/0a8fda330de5436295ce293f2b75fc76.jpg"}
 ...
]


Fix system taints missing

Problem:
Node taints syncer will remove node-role.kubernetes.io/etcd and
node-role.kubernetes.io/controlplane taints. It is because we set
taints both in RKE and Rancher side and the Rancher side will override
those taints added by RKE
Solution:

Apply taints from node pool and node template to RKE config node so RKE
will add them when generating node.
Linux node taints controller for Windows cluster add our taints to v1.Node
directly so we don't need to reuse the taints syncer logic of v3.Node.

Related issue:
https://github.com/rancher/rancher/issues/20805

Updated

No, we dont' deploy windows k8s workloads as part of rke. All of the k8s add-ons we need in windows has the specific scheduling rule to schedule into windows node. And the network plugins in windows won't run as a workload but as binary because the limitation for windows node.


Bug fix for undefined toggleCodeBlockBtn

Getting rid of:
Uncaught TypeError: Cannot set property 'innerHTML' of undefined
on toggleCodeBlockBtn when it doesn't exist

Hi, thank you for your contribution, but they should be made on the development branch which BTW already has a fix for that.


shutil.rmtree cannot deal with file permissions

at patchew-cli:
455                     log = "N/A. Internal error while reading log file\n"
456                 print("  Result:", "Passed" if passed else "Failed")
457                 logging.debug(log)
458                 self.api_do("testing-report", project=r["project"],
459                                               identity=r["identity"],
460                                               test=r["test"]["name"],
461                                               tester=args.name,
462                                               head=r["head"],
463                                               base=r["base"],
464                                               passed=passed,
465                                               log=log,
466                                               is_timeout=is_timeout)
467                 logf.close()
468             finally:
469                 shutil.rmtree(wd)
470             return True
shutil.rmtree does not deal with files generate by root. I'm not sure how to deal with this issue, since running patchew with sudo is ugly. As workaround, i add a line to change file permissions.
I have no propose at this moment to fix this in a good way.

If the test script created these files, it probably makes more sense if they get cleaned up there before the script exits. I cannot think of a good way here either..