ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Thru Lake Formation with permissions set properly on the location and the DB and Table I get the following error message. If I add s3_additional_kwargs with the SSE setting and the Key I get the same error. I can run read_sql_query and read_sql_query and get results just fine, but it works because the KMS key is referenced in Athena for my workgroup. If I add a IAM policy, see below, for s3:GetObject on the resource it works.
I should not have to set perms on S3 but instead we should be able to rely on Lake Formation.
Code:
df2 = wr.s3.read_parquet_table(
database='datalake_dev1_edna',
table='wrkr_cmpstn_cdc_test'
)
Error:
ClientError                               Traceback (most recent call last)
 in
6 df2 = wr.s3.read_parquet_table(
7     database='datalake_dev1_edna',
----> 8     table='wrkr_cmpstn_cdc_test'
9 )
10 df2
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/_config.py in wrapper(*args_raw, **kwargs)
415                 del args[name]
416                 args = {**args, **keywords}
--> 417         return function(**args)
418
419     wrapper.doc = _inject_config_doc(doc=function.doc, available_configs=available_configs)
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_read_parquet.py in read_parquet_table(table, database, filename_suffix, filename_ignore_suffix, catalog_id, partition_filter, columns, validate_schema, categories, safe, map_types, chunked, use_threads, boto3_session, s3_additional_kwargs)
790         use_threads=use_threads,
791         boto3_session=boto3_session,
--> 792         s3_additional_kwargs=s3_additional_kwargs,
793     )
794     partial_cast_function = functools.partial(
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_read_parquet.py in read_parquet(path, path_suffix, path_ignore_suffix, ignore_empty, ignore_index, partition_filter, columns, validate_schema, chunked, dataset, categories, safe, map_types, use_threads, last_modified_begin, last_modified_end, boto3_session, s3_additional_kwargs)
629             use_threads=True,
630             boto3_session=boto3_session,
--> 631             s3_additional_kwargs=s3_additional_kwargs,
632         )
633     return _union(dfs=[_read_parquet(path=p, **args) for p in paths], ignore_index=ignore_index)
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_read_parquet.py in _validate_schemas_from_files(paths, sampling, use_threads, boto3_session, s3_additional_kwargs)
120         use_threads=use_threads,
121         boto3_session=boto3_session,
--> 122         s3_additional_kwargs=s3_additional_kwargs,
123     )
124     _validate_schemas(schemas=schemas)
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_read_parquet.py in _read_schemas_from_files(paths, sampling, use_threads, boto3_session, s3_additional_kwargs)
88                     itertools.repeat(_utils.boto3_to_primitives(boto3_session=boto3_session)),  # Boto3.Session
89                     itertools.repeat(s3_additional_kwargs),
---> 90                     itertools.repeat(use_threads),
91                 )
92             )
~/anaconda3/envs/python3/lib/python3.6/concurrent/futures/_base.py in result_iterator()
584                     # Careful not to keep a reference to the popped future
585                     if timeout is None:
--> 586                         yield fs.pop().result()
587                     else:
588                         yield fs.pop().result(end_time - time.monotonic())
~/anaconda3/envs/python3/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
430                 raise CancelledError()
431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
433             else:
434                 raise TimeoutError()
~/anaconda3/envs/python3/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
382     def __get_result(self):
383         if self._exception:
--> 384             raise self._exception
385         else:
386             return self._result
~/anaconda3/envs/python3/lib/python3.6/concurrent/futures/thread.py in run(self)
54
55         try:
---> 56             result = self.fn(*self.args, **self.kwargs)
57         except BaseException as exc:
58             self.future.set_exception(exc)
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_read_parquet.py in _read_parquet_metadata_file(path, boto3_session, s3_additional_kwargs, use_threads)
54         s3_block_size=131_072,  # 128 KB (128 * 2**10)
55         s3_additional_kwargs=s3_additional_kwargs,
---> 56         boto3_session=boto3_session,
57     ) as f:
58         pq_file: Optional[pyarrow.parquet.ParquetFile] = _pyarrow_parquet_file_wrapper(source=f)
~/anaconda3/envs/python3/lib/python3.6/contextlib.py in enter(self)
79     def enter(self):
80         try:
---> 81             return next(self.gen)
82         except StopIteration:
83             raise RuntimeError("generator didn't yield") from None
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_fs.py in open_s3_object(path, mode, use_threads, s3_additional_kwargs, s3_block_size, boto3_session, newline, encoding)
618             boto3_session=boto3_session,
619             encoding=encoding,
--> 620             newline=newline,
621         )
622         if "b" in mode:  # binary
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_fs.py in init(self, path, s3_block_size, mode, use_threads, s3_additional_kwargs, boto3_session, newline, encoding)
266                 use_threads=False,
267                 boto3_session=self._boto3_session,
--> 268                 s3_additional_kwargs=self._s3_additional_kwargs,
269             )[path]
270             if size is None:
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_describe.py in size_objects(path, use_threads, s3_additional_kwargs, boto3_session)
185     """
186     desc_list: Dict[str, Dict[str, Any]] = describe_objects(
--> 187         path=path, use_threads=use_threads, boto3_session=boto3_session, s3_additional_kwargs=s3_additional_kwargs
188     )
189     size_dict: Dict[str, Optional[int]] = {k: d.get("ContentLength", None) for k, d in desc_list.items()}
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_describe.py in describe_objects(path, use_threads, last_modified_begin, last_modified_end, s3_additional_kwargs, boto3_session)
117     if len(paths) == 1:
118         resp_list = [
--> 119             _describe_object(path=paths[0], boto3_session=boto3_session, s3_additional_kwargs=s3_additional_kwargs)
120         ]
121     elif use_threads is False:
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/s3/_describe.py in _describe_object(path, boto3_session, s3_additional_kwargs)
30         extra_kwargs = {}
31     desc: Dict[str, Any] = _utils.try_it(
---> 32         f=client_s3.head_object, ex=client_s3.exceptions.NoSuchKey, Bucket=bucket, Key=key, **extra_kwargs
33     )
34     return path, desc
~/anaconda3/envs/python3/lib/python3.6/site-packages/awswrangler/_utils.py in try_it(f, ex, ex_code, base, max_num_tries, **kwargs)
324     for i in range(max_num_tries):
325         try:
--> 326             return f(**kwargs)
327         except ex as exception:
328             if ex_code is not None and hasattr(exception, "response"):
~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
355                     "%s() only accepts keyword arguments." % py_operation_name)
356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
358
359         _api_call.name = str(py_operation_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
674             error_code = parsed_response.get("Error", {}).get("Code")
675             error_class = self.exceptions.from_code(error_code)
--> 676             raise error_class(parsed_response, operation_name)
677         else:
678             return parsed_response
ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden
IAM Policy:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:Encrypt",
"kms:ReEncryptTo",
"kms:GenerateDataKey",
"kms:DescribeKey",
"kms:ReEncryptFrom",
"s3:GetObject"
],
"Resource": [
"arn:aws:kms::XXXXXXXXXX:key/",
"arn:aws:s3:::datalake-us-east-1-dev1-curated-secure/*"
]
}
]
}

I am not sure I understand what you mean by "rely on Lake Formation"? Wrangler operations are agnostic to whether Lake Formation is enabled or not. It only relies on the permissions of the IAM role used to make the calls. If the role is missing permissions (like s3:HeadObject in your case), it's normal that the call will fail.

Lake Formation uses credential vending, see below. In my case I am running a SageMaker notebook that uses a role that had been granted permissions in Lake Formation on the DB, Table, and Location. Lake Formation simplifies data lake management we would not want to start granting direct S3 access. I am a AWS Proserve DataLake Architect working for a large airline. We can work around this by providing direct access but longer term would want to rely on Lake Formation to manage perms.
Each time a principal (user, group, or role) runs a query on data registered using Lake Formation, Lake Formation verifies that the principal has the appropriate Lake Formation permissions to the database, table, and Amazon S3 location registered in Lake Formation as appropriate for the query. If the principal has access, Lake Formation vends temporary credentials, and the query runs.
The following diagram illustrates the flow described above.

Closing as this level of Lake Formation support is not currently on the roadmap


Snapshot Visibility Bug

Win7 x64
SWS 2.8.0 build 0
Problem: In certain situations visibility snapshots are saving/recalling custom filter options instead of track visibility.
Steps to reproduce:

Open Snapshots and select any custom filter options
Use action _SWSSNAPSHOT_VISMODE to switch to visibility mode
Save new snapshot

Expected:  I've saved a new visibility snapshot
Actual Result:  I have actually saved a new custom snapshot with most recent custom settings
Why it's an issue:  I've been using cycle actions to switch between different sets of visible tracks.  Recently I noticed recalling these snapshots were sometimes undoing extensive mixing work instead of changing track visibility.  Finally noticed that it was because my "visibility" snapshots were actually custom snapshots, recalling various mix parameters, even between loading project sessions.

Afaict, both the actions _SWSSNAPSHOT_MIXMODE and _SWSSNAPSHOT_VISMODE are broken.  They appear to change the mode in the Snapshot window, but new snapshots do not use the correct settings, instead using whichever options are checked (and greyed out) in the Filter under "Custom."

Thanks for the detailed report Poofox (that helped!) , should be fixed in the next pre-release


Add hyperlinks in text/pdf file

The links in the text file need copy-pasting in the browser for viewing the articles. The possible outcome should be to add hyperlinks in the text file.

I have been working on adding the PDF option and hyperlinks in the output file.
Will send a PR soon


Parse Error when inserting records into tables in many-to-many relationship

Environment
Knex version: 0.15.2
Database + version: MariaDB 10.1.37
OS: Antergos (Arch Linux)
Select applicable tempalate from below.
If issue is about oracledb support tag @ atiertant. For MSSql tag @ smorey2 .
Rest of dialects doesn't need tags.
Bug

Explain what kind of behaviour you are getting and how you think it should do
I have 2 tables (users, permissions) which are linked over a junction table (user_permission). I'm trying to insert a record into the tables with a raw query with Knex but I'm getting a SQL Parse Error.  The same queries:

START TRANSACTION;
INSERT INTO permissions(name) VALUES('Permission 1');
SET @last_permission_id = LAST_INSERT_ID();
SET @user_id = (SELECT id FROM users WHERE last_name LIKE '%Smith%');
INSERT INTO user_permission (user_id, permission_id) VALUES (@user_id, @last_permission_id);
COMMIT;

work fine when I test them in Adminer.

Error message

Unhandled rejection Error: ER_PARSE_ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MariaDBserver version for the right syntax to use near 'INSERT INTO permissions(name) VALUES('Something19');
SET @last_permission_id = L' at line 2
    at Query.Sequence._packetToError (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
    at Query.ErrorPacket (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/protocol/sequences/Query.js:77:18)
    at Protocol._parsePacket (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/protocol/Protocol.js:278:23)
    at Parser.write (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/protocol/Parser.js:76:12)
    at Protocol.write (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/protocol/Protocol.js:38:16)
    at Socket.<anonymous> (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/Connection.js:91:28)
    at Socket.<anonymous> (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/Connection.js:502:10)
    at Socket.emit (events.js:182:13)
    at addChunk (_stream_readable.js:287:12)
    at readableAddChunk (_stream_readable.js:268:11)
    at Socket.Readable.push (_stream_readable.js:223:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:122:17)
    --------------------
    at Protocol._enqueue (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/protocol/Protocol.js:144:48)
    at Connection.query (/home/user/workspace/ExpressJS/test_project_02/node_modules/mysql/lib/Connection.js:200:25)
    at /home/user/workspace/ExpressJS/test_project_02/node_modules/knex/lib/dialects/mysql/index.js:154:18
    at Promise._execute (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/debuggability.js:313:9)
    at Promise._resolveFromExecutor (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/promise.js:483:18)
    at new Promise (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/promise.js:79:10)
    at Client_MySQL._query (/home/user/workspace/ExpressJS/test_project_02/node_modules/knex/lib/dialects/mysql/index.js:148:12)
    at Client_MySQL.query (/home/user/workspace/ExpressJS/test_project_02/node_modules/knex/lib/client.js:202:17)
    at Runner.<anonymous> (/home/user/workspace/ExpressJS/test_project_02/node_modules/knex/lib/runner.js:146:36)
    at Runner.tryCatcher (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/util.js:16:23)
    at Runner.query (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/method.js:15:34)
    at /home/user/workspace/ExpressJS/test_project_02/node_modules/knex/lib/runner.js:48:21
    at tryCatcher (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/util.js:16:23)
    at /home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/using.js:185:26
    at tryCatcher (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/home/user/workspace/ExpressJS/test_project_02/node_modules/bluebird/js/release/promise.js:512:31)

Reduced test code, for example in https://npm.runkit.com/knex or if it needs real
database connection to mysql or postgresql, then single file example which initializes
needed data and demonstrates the problem.

var myquery = `START TRANSACTION;
INSERT INTO permissions(name) VALUES('Permission 1');
SET @last_permission_id = LAST_INSERT_ID();
SET @user_id = (SELECT id FROM users WHERE last_name LIKE '%Smith%');
INSERT INTO user_permission (user_id, permission_id) VALUES (@user_id, @last_permission_id);
COMMIT;`

knex.raw(myquery).then(function (records) {
    console.log(JSON.parse(JSON.stringify(records[0])));
});

Try to use mariadb driver directly to run that query and report back if it works there.  Nothing in this report suggests that there is a bug in knex causing this.


Remove committers section?

In the generated Dockerfile header I have the authors. Thinking this can be removed OR just use 'https://github.com/AnsibleShipyard' as the author. I'm leaning toward the link. Thoughts?

The link sounds good to me


Request: API-only artifacts published to Maven

See JEI for an example:
task apiJar(type: Jar) {
	from sourceSets.main.allSource
	from sourceSets.main.output
	include 'mezz/jei/api/**/*'
	classifier = 'api'
}

artifacts {
	archives apiJar
}

Would allow mods to add support for TOP without having to add TOP itself to the dev environment.

TOP already has a maven here:
http://ci.tterrag.com/job/TheOneProbe/
Just use that in your build.gradle and you will be fine. TOP (and the API) will be available automatically in your dev env

You seem to have misunderstood--I don't want to include the full mod in the dev environment, I only want the API package. Again, see JEI for an example:
http://dvs1.progwml6.com/files/maven/mezz/jei/jei_1.10.2/<IP_ADDRESS>/jei_1.10.2-<IP_ADDRESS>-api.jar
It only includes the files in the mezz/jei/api directory, which allows you to successfully import and compile against the API, but will not add the mod itself to the environment.
I do the same thing for AppleCore: https://github.com/squeek502/AppleCore/wiki/Building-against-AppleCore

Why would you want that? That way you cannot test your API work?

I could then add and remove TOP from the dev environment at will by including the TOP jar in the mods directory as normal.

What exactly is the problem with having TOP in the dev env all the time? I always have WAILA, TOP, JEI and a few other mods in all my dev envs constantly

It's a personal preference, but I think it has a few benefits:

Increases the potential for conflicts/bugs/etc to not be found because they are covered up by the mod always being in the dev environment (for example, something that is intended to be optional, in this case TOP support, not actually being optional)
Reduces potential issues when setting up a new dev environment
Reduces loadtime of a fresh dev environment
Increases flexibility of what mods you have included in your dev environment

Including the full mod is totally fine when making an addon, but that's not what I'm doing. I just want to add optional support and this use case is exactly why the API package exists.

I'm a noob on things related to maven but can you explain to me more how exactly to set this up so it actually publishes fine on the maven that I already have? And how would one then use this instead of the current way? i.e. what does one have to use in build.gradle to get only the api and not the full package?

Sure, it should be pretty easy. I've never use the maven-publishing plugin, but from what I can tell, you'd just need to add this after this line:
artifact apiJar

and add this somewhere in build.gradle:
task apiJar(type: Jar) {
    from sourceSets.main.output
    include 'mcjty/theoneprobe/api/**'
    classifier = 'api'
}

artifacts {
    archives apiJar
}

The new API artifact could then be included in a dev environment by doing:
deobfCompile "mcjty.theoneprobe:TheOneProbe:1.10-1.1.4-35:api"

instead of
deobfCompile "mcjty.theoneprobe:TheOneProbe:1.10-1.1.4-35"

Thanks for doing this @McJty.


File icons are all missing after the recent insider update

VSCode Version: 1.32-insider
OS Version: Mojave

Steps to Reproduce:

Update the insider version
All file icons disappear

Does this issue occur when all extensions are disabled?: Yes/No

I'm on
Version: 1.32.0-insider
Commit: 9441f43636a639aaffa28a9c7e6e36151944c8b3
Date: 2019-02-12T06:13:34.951Z
Electron: 3.1.3
Chrome: 66.0.3359.181
Node.js: 10.2.0
V8: 6.6.346.32
OS: Linux x64 4.15.0-45-generic
and I still see file icons.
Also I tested that the update does not make changes to the setting.
Please explain what you exactly see. Are the icon themes still available?

It happens for both my windows machine at home and Mac at work.
Could be the material theme I'm using. I'll open another ticket in the related repo. Might just be a bad timing if you can't reproduce it. Thanks.

Resolved. It's because the icons in the theme were split into a dedicated extension. Please close this issue. Thanks.

@pingshunhuang Are you able to detail the steps to resolve this issue? I am having similar problems after using the material themes. Thanks :+1:

@pingshunhuang Brilliant, that worked thanks ! :+1:


daft.exceptions.ByteStreamError: Unable to read data from file s3

Describe the bug
A clear and concise description of what the bug is.
Following code executes
df = daft.from_glob_path(
"s3://bucket/*ipname.json.gz"
)
df.collect()
df = df.with_column("data", df["path"].url.download())
df.collect()
and I sometimes get following message (sometimes code works)
Project [Stage:2]:   0%|          | 0/1 [00:00<?, ?it/s]Traceback (most recent call last):
File "/app/output_parquet.py", line 89, in
df.collect()
File "/usr/local/lib/python3.9/site-packages/daft/api_annotations.py", line 26, in _wrap
return timed_method(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/daft/analytics.py", line 189, in tracked_method
result = method(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/daft/dataframe/dataframe.py", line 1466, in collect
self._materialize_results()
File "/usr/local/lib/python3.9/site-packages/daft/dataframe/dataframe.py", line 1448, in _materialize_results
root@e7c8865b50f3:/# exit
^Mexit
...skipping...
self._result_cache = context.runner().run(self._builder)
File "/usr/local/lib/python3.9/site-packages/daft/runners/pyrunner.py", line 135, in run
results = list(self.run_iter(builder))
File "/usr/local/lib/python3.9/site-packages/daft/runners/pyrunner.py", line 187, in run_iter
yield from results_gen
File "/usr/local/lib/python3.9/site-packages/daft/runners/pyrunner.py", line 279, in _physical_plan_to_partitions
materialized_results = done_future.result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/daft/runners/pyrunner.py", line 325, in build_partitions
partitions = instruction.run(partitions)
File "/usr/local/lib/python3.9/site-packages/daft/execution/execution_step.py", line 438, in run
return self._project(inputs)
File "/usr/local/lib/python3.9/site-packages/daft/execution/execution_step.py", line 442, in _project
return [input.eval_expression_list(self.projection)]
File "/usr/local/lib/python3.9/site-packages/daft/table/micropartition.py", line 169, in eval_expression_list
return MicroPartition._from_pymicropartition(self._micropartition.eval_expression_list(pyexprs))
daft.exceptions.ByteStreamError: Unable to read data from file s3://bucket/path.json.gz: error reading a body from connection: end of file before message length reached
To Reproduce
Steps to reproduce the behavior:
Execute code to download files from s3 and sometimes error above occurs
Expected behavior
Sometimes the code works and downloads json.gzip data
Desktop (please complete the following information):
Mac Sonoma 14.4.1 (23E224)

Hi @alexmi, thanks for making this issue!
We've seen this happen when Daft is too aggressive with downloading from S3, and S3 starts cutting our connections 🤯 . It sometimes just depends on how well S3 is scaling up to deal with the work that Daft is throwing at it.
Could you give us an idea of how many files you reading?

Hi @jaychia,
Thank you for your reply.  I have 148 files total match the glob pattern and url.download is applied. Once the files are unzipped and converted to pandas/parquet it saves to one file of 168Mb.
Is there a way to specify number of retries similar to boto3 ?
Thanks

Our url download docs also has some suggestions:

If you are observing excessive S3 issues (such as timeouts, DNS errors or slowdown errors) during URL downloads, you may wish to reduce the value of max_connections (defaults to 32) to reduce the amount of load you are placing on your S3 servers.

Try this!
df["urls"].url.download(max_connections=8)

That fixes the problem. My bad to not finding the docs. I'll close this.

Sounds great @alexmi !
I'm also curious - how many cores does your machine have? We're seeing issues around BytesStream interruptions in other I/O code as well.

@jaychia  I was running it on 2.6 Ghz 6-Core Intel Core i7 2019 mac but in a Docker that had 6 CPU setting.  Not sure if it is equal to 6 cores or less.  Perhaps default of max_connections should be tied to number of CPUs.


Feature: "fixed" prop regardless of "maxSuggestions"

I'm using AutoCompleteGroup to show a divider then a fixed custom item (eg. Show All Categories) that should always be at the end of the suggestions -- however when limiting suggestions to a small number the fixed item doesn't show if the suggestions are maxed out.
Solution ideas:

Make fixed item show even when suggestions reach maxSuggestions.
Or
Include maxSuggestions prop for individual AutoCompleteGroup that would apply only to that grouping.
Or
Make a separate alwaysShow item that's present regardless of 'maxSuggestions`.

Attached gif shows the AutoCompleteItem ("Show All Categories") I have after an AutoCompleteGroup that is intended to show after the regular suggestions, however it only shows if total suggestions < maxSuggestions={3}

Hi @daliudzius Could you create a codesandbox reproduction of this? thanks

@anubra266 Sure, here you go: https://codesandbox.io/s/dazzling-mclean-pkqxw?file=/src/with-icon.js
Usage

If you type "d" it displays the 2 suggestions + the fixed item that is intended to always be in the suggestions regardless of maxSuggestions limit.
If you type "a" it displays 3 suggestions according to maxSuggestions however does not take into account the fixed item which would be at the end of the array of suggestions.

Additional notes
I created a pull request you with an idea of adding another prop alwaysShow which would behave similar to fixed, although it would show regardless of both filter and maxSuggestions.

@daliudzius i think it will be better if we tweak fixed to always show instead of adding a new prop that aims to do the same thing.

Agreed, @anubra266. I was hesitant to suggest that in the event that I was missing some use case for fixed to function within the scope of maxSuggestions as it currently does.


BREAKING CHANGE: rename Source class to PushSource

Proposed changes
Do not use the generic Source keyword as the SDK will also be supporting Catalog sources.
This:
const source = new Source(...)

will eventually be:
const push = new PushSource(...)
const catalog = new CatalogSource(...)

Breaking changes
Source class has been deprecated.
Testing

[ ] Unit Tests:

[ ] Functionnal Tests:

[ ] Manual Tests:

If that was the CLI, this would not be a problem, but this is a library, so logging is not something we do willy nilly IMO.
However, I don't think honestly we have enough experience in the 'library development field' to take an educated decision.
So, in that case, I think you should research what are the good practices on the field so that we can come to a conclusive and educated decision on this matter.
This might seems overkill, but the conclusion tha we reach today also something that will guide us in the future design of our SDKs/libraries, so I think personally that it is worth it.

I agree about leaving the logging up to the application using the SDK. That way, users can easily control and centralize logs altogether.
With that said, these console.logs are not about debugging nor capturing information about the execution result.
If we were to implement a more advanced logging system by allowing users to provide there own log function/callback, they would probably miss the warning (since the log function or callback would be optional), which would go against the entire idea of making sure users know when they are using a deprecated API.


[SPARK-42038][SQL] SPJ: Support partially clustered distribution

What changes were proposed in this pull request?

Currently with storage-partitioned join, both sides of the join must be fully clustered over the partition values, that is, each Spark partition should have a distinct partition value. To guarantee this, Spark will group all the input partitions reported by a V2 data source on the partition values. The consequence, however, is that it can easily lead to data skew, when a few partition values have large amount of data.
This PR introduce a new mechanism which requires only one side of the storage-partitioned join to be fully clustered, while the other side can be partially clustered, i.e., there could exist multiple Spark partitions with the identical partition value. At planning time, Spark will compare the statistics from both sides of the join, and pick the side with smaller size as full clustered side, while the other side is partially clustered. It then replicate the partitions on the former to match the number of partitions on the latter.
A new config spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled is added to enable & disable the feature.
For instance, consider the following SQL query:
SELECT * FROM a JOIN b on a.id = b.id

while table a reports partitions [0, 1, 2] while b reports partitions [0, 1, 1, 1, 2, 2].
Without the PR, Spark would group input partitions so that both sides will have 3 partitions [0, 1, 2]. With the PR, Spark will choose the right-hand side as partially clustered and match the left-hand side with it. Therefore, both sides have 6 partitions [0, 1, 1, 1, 2, 2].
Note this PR currently relies on a simple heuristic and always pick the side with less data size based on table statistics as the side fully clustered, even though it could also contain skewed partitions. In future, we can potentially do fine-grained comparison based on partition values.
Why are the changes needed?

As mentioned in the previous section, this feature can help to reduce data skewness when a partition value is mapped to a large amount of rows.
Does this PR introduce any user-facing change?

A new Spark config, spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled, is introduced to enable/disable the feature. By default it is disabled, so the behavior should still be the same as before.
How was this patch tested?

Added new tests in KeyGroupedPartitioningSuite.

@sunchao do you plan to add this for 3.4? cc @xinrong-meng FYI

@HyukjinKwon it'd be great if this can make it into 3.4, but it's not a release blocker.

Are we replicating the AQE skew join optimization in EnsureRequirements? I'm a bit worried about direction as EnsureRequirements is becoming more and more complicated.

Since the storage partitioned join becomes more and more complicated, shall we revisit the framework? Essentially we need to join the partitions from two v2 tables at the driver side, and launch a task to process each partition pair. One idea is to add a physical rule before EnsureRequirements. This new rule matches join of two v2 tables, applies a join strategy, and sets a special flag in the join node so that EnsureRequirements can skip it. This is similar to ShuffledJoin.isSkewJoin:
trait ShuffledJoin extends JoinCodegenSupport {
  def isSkewJoin: Boolean

    override def requiredChildDistribution: Seq[Distribution] = {
    if (isSkewJoin) {
      // We re-arrange the shuffle partitions to deal with skew join, and the new children
      // partitioning doesn't satisfy `HashClusteredDistribution`.
      UnspecifiedDistribution :: UnspecifiedDistribution :: Nil
    } else {
      ClusteredDistribution(leftKeys) :: ClusteredDistribution(rightKeys) :: Nil
    }
  }
}

@cloud-fan the idea is similar to skew join but for v2 sources, let me try to split the code into a separate rule following your idea.

Hmm @cloud-fan , it seems a bit tricky to split the logic specific to SPJ out of EnsureRequirements, since it depends on reorderJoinPredicates, which is currently co-mingled with EnsureRequirements. I was trying to split reorderJoinPredicates into a separate rule but just realized I hit SPARK-22042.

Shall we convert to Ready for review status back, @sunchao ?

Yes, converted.
I found it is quite difficult to move the logic out of EnsureRequirements, because, as mentioned above, the optimization also depends on reorderJoinPredicates which needs to be executed together with ensureDistributionAndOrdering in a lock-step as we go up the query plan.
In addition, even if I extract out reorderJoinPredicates into a separate util method or something, it's still difficult to make the optimization as a separate rule because it also depends on certain part of the logic in ensureDistributionAndOrdering (for instance, as we go up the query plan tree, we'd expect join branches that have incompatible KeyGroupedPartitionings to be handled by ensureDistributionAndOrdering and converted to use hash partitioning, so that we should not try to apply the optimization again on those branches).
As a compromise, I've extracted all special logic related to KeyGroupedPartitioning into a separate method checkKeyGroupCompatible, so the main body of ensureDistributionAndOrdering now looks much simpler. If necessary, I can also move this method (and related) into a separate class file instead, to make them more isolated.

Thank you, @sunchao !

Could you resolve the conflicts once more, @sunchao ?

Merged to master/3.4.
Thank you so much, @sunchao , @cloud-fan , @viirya .
Also, cc @aokolnychyi and @RussellSpitzer

Thanks @cloud-fan @dongjoon-hyun @viirya !


scss: bring light/dark accordion icon color handling to parity

The light themed accordion icon is controlled by a set of two variables:

$accordion-icon-color
$accordion-icon-active-color

The dark themed accordion icon did not have this level of control available and instead would always use $primary-text-emphasis-dark.
Replicate the behaviour to permit individual control over the colours to match the behaviour for the light themed form.
Description
The dark themed accordion does not permit control over the icon colour that the light themed accordion allows.
Motivation & Context
Changing the color of the icon for the dark theme is currently not possible which can cause trouble if the emphasis text colour is desired to be different from the chevron on the accordion.
Type of changes

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Refactoring (non-breaking change)
[ ] Breaking change (fix or feature that would change existing functionality)

Checklist

[x] I have read the contributing guidelines
[x] My code follows the code style of the project (using npm run lint)
[ ] My change introduces changes to the documentation
[ ] I have updated the documentation accordingly
[ ] I have added tests to cover my changes
[x] All new and existing tests passed

Live previews

https://deploy-preview-{your_pr_number}--twbs-bootstrap.netlify.app/

Related issues

Just to verify, this is not blocked on anything else needed from me, correct?

Thanks for the PR @compnerd
I've just approved the workflows. It's not enabled when it's your first contribution.
The deployed version of this branch/PR will be in https://deploy-preview-38279--twbs-bootstrap.netlify.app/.
The team will try to review your PR asap


messageBubbleFont != nil

IFF I present my JSQMessagesViewController modally when the app is opened from suspended, I get the following crash:
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid parameter not satisfying: messageBubbleFont != nil'
In viewDidLoad, I have         self.collectionView.collectionViewLayout.messageBubbleFont = [UIFont systemFontOfSize:14 weight:UIFontWeightLight];
I never set it to be nil...
Any ideas?

Not setting self.collectionView.collectionViewLayout.springinessEnabled = BOOL in viewDidAppear stopped the crashing!
This is why I should have read the fine print!!! (Note: this feature is mostly stable, but still experimental)

Thanks all! :smile:
Now tracking this at #1338.


ci: ignores flake8-bugbear B905

flake8-bugbear 22.12.6 introduces a new rule v905 that can be fixed only
if we requires python >= 3.10. Until we requires >= 3.7, we can't enable
this rule.
This change ignores B905 flake8-bugbear rule.

Please target this at the current release branch, 3.1.x.

Also, the same issue affects Werkzeug, Click, MarkupSafe, and ItsDangerous, if you want to make more PRs.


update jhipster to 7.9.3

Update jhipster version to 7.9.3

Failures looks related to https://github.com/jhipster/generator-jhipster/pull/19301.
Windows failures are related to npmw.cmd changes.

Will have a look

At windows, we should run with —skip-install and run npm install later, I can take a look later.

I will try to have a look at the oauth part this evening. JWT seems to work fine indeed.

It made it work with caffeine. Right now this will only support caffeine directly for the oauth caching. Spring cache integration seems not to work, so our recommendation, to run this without a cache provider still stands imho.

At windows, we should run with —skip-install and run npm install later, I can take a look later.

@mshima Can you give me hint where to look at? It is something we should configure in the ci pipeline or in generated code?

@atomfrede sure.
It's related to https://github.com/jhipster/generator-jhipster/pull/19404.
GraalVM adds a lot of paths to $PATH, https://github.com/jhipster/generator-jhipster/pull/19404 added more path overflowing the max size of $PATH.
The workaround is to execute jhipster-native jdl command with --skip-install and run npm install later.

I will do the initial change.

Thanks. I remember that with bootrun and gradle some time ago

Looks good. Failing macos tests fail to install docker in the pipeline, so I would say good to merge imho. Thanks @mshima for doing the windows ci part!

Thanks @atomfrede


Unify usage of settings and remove the usage of getattr for settings

I wasn't quite sure what to with this so I left it as it is:
https://github.com/rtfd/readthedocs.org/blob/7fa9002586b2c5c2fae5c78e5ad736d9d46f7ad4/readthedocs/core/utils/extend.py#L29
closes #2140

@stsewd Updated the PR with the changes you suggested :)

I just wanted to chime in that I'm excited for this change. This will simplify a lot of things and centralize all our settings in one place so they are easier to understand!

I wasn't quite sure what to with this so I left it as it is:
https://github.com/rtfd/readthedocs.org/blob/7fa9002586b2c5c2fae5c78e5ad736d9d46f7ad4/readthedocs/core/utils/extend.py#L29

That should remain as it is.

We still to change more code

@stsewd I might have missed these two from my search. Updated the PR!

This PR touches many different files. I'm merging it now to avoid conflicts and also to start testing it locally considering that there are many settings re-defined. Although, the defaults on base.py looks accurate and the same as the ones used by getattr. I'm not expecting to fail on prod, but we should be careful when deploying it.

I think we should probably do a followup here to remove places where we're using sync_user = settings.SYNC_USER to use the setting directly. We're doing this a number of places in the code, and I think it's a bad pattern.

This is fine for now though 👍

I think we should probably do a followup here to remove places where we're using sync_user = settings.SYNC_USER to use the setting directly. We're doing this a number of places in the code, and I think it's a bad pattern.

@ericholscher Okay! I will start working on it:)

I think we should probably do a followup here to remove places where we're using sync_user = settings.SYNC_USER to use the setting directly. We're doing this a number of places in the code, and I think it's a bad pattern.

@ericholscher @saadmk11 I already mentioned this at https://github.com/rtfd/readthedocs.org/pull/5588#discussion_r275119621
I think the new PR should include all the changes for this pattern.


RequiresAuthentication in OWIN

I see that there is a method to require authentication if you are using Nancy.  Is there similar functionality in OWIN?

No, at the moment authentication has not yet been implemented for the OWIN adapter.
Iulian, looks like you have started something for the Nancy adapter, how are you planning to go about authentication?

@alhardy I really don't want to do any authentication/security related functionality directly in the Metrics library. Metrics library should easily integrate with any existing frameworks/applications/libraries that provide authentication and authorization functionality.
The Nancy method for authentication is provided by Nancy and not by the Metrics library. The Nancy adapter in the Metrics library just provides an easy way to the user to call the auth related methods exposed by Nancy.
As for Owin, I'm not an expert, but it is my understanding that the whole point of Owin is to be able to define and compose independent middlewares each providing different functionality. If I'm not mistaken then authentication should be handled by an authentication middleware.
A quick google check seems to confirm that:
http://blogs.msdn.com/b/webdev/archive/2013/07/03/understanding-owin-forms-authentication-in-mvc-5.aspx
http://coding.abel.nu/2014/06/writing-an-owin-authentication-middleware/
If you have any issues integrating Metrics with such middleware implementation do let us know.

@jfritchman @alhardy i'm closing this for now, but feel free to re-open if more information can be added.


1.35.5 - INVALID CHECKSUMS produced by Jfrog CLI - Artifactory reports 409 because of it

Using jfrog cli 1.35.5 to publish to Artifactory reports incorrect checksums and returns 409
Using Jfrog cli 1.32.4 to publish gradle builds to Artifactory, works as expected.
$ md5sum api-security-1.0.19-SNAPSHOT.jar
e342f9c0f633a8f3963aff4a52f29d2a api-security-1.0.19-SNAPSHOT.jar

$ sha1sum api-security-1.0.19-SNAPSHOT.jar
a599acbb955b933dfbdc2d1f684061fbad43a5e9 api-security-1.0.19-SNAPSHOT.jar

$ shasum -a256 api-security-1.0.19-SNAPSHOT.jar
c289f8d3618357897e3d8c500a74676734d85169d3647fae23f70291e40528b0 api-security-1.0.19-SNAPSHOT.jar

Artifactory 409 error reports this:

{
checksums = {
SHA - 1 = ChecksumInfo

{ type = SHA - 1, original = '26c15a328950d10f1cd75a5eeb9108c38b3147bd', actual = 'a599acbb955b933dfbdc2d1f684061fbad43a5e9' }
,
MD5 = ChecksumInfo

{ type = MD5, original = 'e629e280cb3b84fae1da87557ef0dec1', actual = 'e342f9c0f633a8f3963aff4a52f29d2a' }
,
SHA - 256 = ChecksumInfo

{ type = SHA - 256, original = 'null', actual = 'c289f8d3618357897e3d8c500a74676734d85169d3647fae23f70291e40528b0' }
}
}

Example response:
> Task :artifactoryDeploy FAILED
 FAILURE: Build failed with an exception.
 * What went wrong:
 Execution failed for task ':artifactoryDeploy'.
 > java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: Failed to deploy file. Status code: 409 Response message: Artifactory returned the following errors:
   Checksum policy 'LocalRepoChecksumPolicy: CLIENT' rejected the artifact 'libs-snapshot-local:com/ur-test/api-security/1.0.19-SNAPSHOT/api-security-1.0.19-SNAPSHOT.jar'. Checksums info: ChecksumsInfo{checksums={SHA-1=ChecksumInfo{type=SHA-1, original='26c15a328950d10f1cd75a5eeb9108c38b3147bd', actual='a599acbb955b933dfbdc2d1f684061fbad43a5e9'}, MD5=ChecksumInfo{type=MD5, original='e629e280cb3b84fae1da87557ef0dec1', actual='e342f9c0f633a8f3963aff4a52f29d2a'}, SHA-256=ChecksumInfo{type=SHA-256, original='null', actual='c289f8d3618357897e3d8c500a74676734d85169d3647fae23f70291e40528b0'}}} Status code: 409

@alexmnyc thanks for opening this issue.
Checksum mismatch can sometimes be caused by network issues, is this error consistent?
As I wasn't able to reproduce the issue on my machine:

Is this also happening in other gradle builds? Can you please try building our gradle project example and share the outcome?
Which version of Artifactory do you use?
What is the CLI command you run?

The error is consistent for the same codebase on different CI instances and publishing to two different artifactories
Artifactory version is 6.18.1 rev 61801900

Is original value what Jfrog clients calculates and sends it to artifactory? That value seems to be incorrect
It works for some builds, especially with gradle 5 6 etc
This specific build is using gradle wrapper 4.10
Shadow plugin 4.0.3

@alexmnyc,
Can you please help us reproduce this issue? Maybe share a simple gradle project in GutHub that shows this issue.
Thanks!

The code is proprietary as I work at TRP. Scott, on our team, opened a ticket with you guys internally I believe 134708

I work with @alexmnyc and can report that JFrog CLI 1.36.0 does not exhibit this behaviour, whereas 1.35.5 does.
There's nothing in the release notes which looks relevant and nothing in the commit history that jumps out at me either.
jfrog-client-go has been updated from 0.9.0 to 0.11.0 but there's nothing in the release notes for that which seems relevant
build-info-gradle-extractor has been updated from 4.15.0 to 4.15.2 - I wonder if this PR is the cause for the fixed behaviour: https://github.com/jfrog/build-info/pull/336
As an aside I see that the default environment variable excludes have been changed with 1.36.0 so that environment variables with "psw" in their name are no longer published with buildinfo, but this is not mentioned in the release notes. It doesn't affect us but seems worth pointing out.

Thanks for the update @rhowe!
We will update the release notes with the psw exclude addition. Sorry for missing that.
As for the actual checksum issue, this still remains a mystery. I can't think of anything right now that can cause this. Please let me know if you'd like us to continue working together to figure this one out. If not, feel free to close this issue for now.

Closing this issue for now @rhowe. Please let me know if you'd like to continue looking into this together.


feat: luarocks/rocks.nvim support

Hey :wave:
Summary
This PR is part of a push to get neovim plugins on LuaRocks.
See also:

rocks.nvim, a new luarocks-based plugin manager.
this blog post.

With luarocks/rocks.nvim, it is the plugin authors' responsibility to declare dependencies - not the user's.
Installing a plugin that depends on this one becomes as simple as :Rocks install <plugin>.
Things done:

Add a workflow that publishes tags to luarocks.org when a tag or release is pushed.
Add a section to the readme for plugin developers,
linking to a guide for publishing plugins to luarocks

Notes:

[!IMPORTANT]

For the luarocks workflow to work, someone with a luarocks.org account will have to add their API key to this repo's GitHub actions secrets.

Tagged releases are installed locally and then published to luarocks.org.

If you push tags from a local checkout, the workflow is triggered automatically.
If you use GitHub releases to create tags, you may need to add a PA token for the workflow to be triggered automatically.

Due to a shortcoming in LuaRocks (https://github.com/luarocks/luarocks-site/issues/188), the neovim and/or vim labels have to be added  to the LuaRocks package manually (after the first upload), for this plugin to show up in https://luarocks.org/labels/neovim or https://luarocks.org/labels/vim, respectively.

See also this guide.

thanks @mrcjkb! 💪 I will look into that over the weekend

@mrcjkb all configured! ✅ https://luarocks.org/modules/mobily/nui-components.nvim


Dev/use shutil for copy

This is patch to fix #6003 by using an internal copy rather than a tar/untar mechanism.
This turned out to be an extraordinarily tricky patch to get right because you need to get
all of the permissions and ownerships correct.  Also the patch hard codes certain decisions
for the copy such as to not follow symlinks.

opps patch include some other tings.


update with airbnb prediction and notebook EDA

Add Airbnb prediction notebook and Airbnb dataset to the master branch.

lgtm


[UI] Use High Quality Image on the Homepage

Category

[ ] Resource Addition
[ ] Documentation
[ ] Codebase
[x] User Interface
[ ] Feature Addition

Description
The quality of the image on the Homepage is not good.

@Sriparno08 Assign me this Issue


class EscapeFormatter(string.Formatter): AttributeError: module 'string' has no attribute 'Formatter'

I'm trying to use Jinja2 in my project. Jinja2 internally uses markupsafe, I'm getting the below error. Not sure How this can be fixed. Please let me know if there is a solution for this issue?
  File "C:\Program Files\Python35\lib\site-packages\jinja2\__init__.py", line 33, in <module>
    from jinja2.environment import Environment, Template
  File "C:\Program Files\Python35\lib\site-packages\jinja2\environment.py", line 15, in <module>
    from jinja2 import nodes
  File "C:\Program Files\Python35\lib\site-packages\jinja2\nodes.py", line 19, in <module>
    from jinja2.utils import Markup
  File "C:\Program Files\Python35\lib\site-packages\jinja2\utils.py", line 647, in <module>
    from markupsafe import Markup, escape, soft_unicode
  File "C:\Program Files\Python35\lib\site-packages\markupsafe\__init__.py", line 250, in <module>
    class EscapeFormatter(string.Formatter):
AttributeError: module 'string' has no attribute 'Formatter'

I can't reproduce your issue. Python 3.5's string module definitely has a Formatter class. Did you name a file of your own string.py and mask Python's module?


Goto Def is not working for new when the object is defined in different bal file/module

Description:
The goto the object constructor when referring to new seems to be not working when the object is defined in different bal file/module.
Steps to reproduce:
Try Goto def with new and it doesn't work as expected.
main.bal
import ballerina/io;

public function main() {
    // Initializes a `Person` object.
    Person p1 = new (5, "John", "Doe");

    io:println(p1.getFullName());
}

person.bal
type Person object {
    public int age;
    public string firstName;
    public string lastName;

    // The object initializer.
    function __init(int age, string firstName, string lastName) {
        self.age = age;
        self.firstName = firstName;
        self.lastName = lastName;
    }

    // A method returning the full name value of the `Person` object.
    function getFullName() returns string {
        return self.firstName + " " + self.lastName;
    }
};

Affected Versions:
1.2.0-RC2
OS, DB, other environment details and versions:
MacOS
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

Duplicate of #21790. Hence closing the issue


Add docs and formatting for "encryption-api" module

Add documentation and formatting for the encryption-api module.
Move String -> PublicKey object conversion into similar existing service class.

Codecov Report

Merging #650 into master will increase coverage by 1.37%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #650      +/-   ##
============================================
+ Coverage     98.08%   99.45%   +1.37%
+ Complexity     1915     1914       -1
============================================
  Files           307      300       -7
  Lines          6010     5919      -91
  Branches        281      279       -2
============================================
- Hits           5895     5887       -8
+ Misses           87        5      -82
+ Partials         28       27       -1

Impacted Files
Coverage Δ
Complexity Δ

.../main/java/com/quorum/tessera/nacl/NaclFacade.java
100% <ø> (ø)
3 <0> (ø)
:arrow_down:

.../com/quorum/tessera/encryption/KeyManagerImpl.java
100% <100%> (ø)
11 <1> (ø)
:arrow_down:

...ava/com/quorum/tessera/enclave/EnclaveFactory.java
100% <100%> (ø)
7 <0> (ø)
:arrow_down:

...i/src/main/java/com/quorum/tessera/nacl/Nonce.java
100% <100%> (+14.28%)
7 <1> (+1)
:arrow_up:

...rum/tessera/keypairconverter/KeyPairConverter.java
100% <100%> (ø)
7 <5> (+2)
:arrow_up:

...rum/tessera/config/util/PasswordReaderFactory.java

...va/com/quorum/tessera/enclave/websockets/Main.java

...java/com/quorum/tessera/config/migration/Main.java

...n/java/com/quorum/tessera/data/migration/Main.java

...rum/tessera/config/util/ConsolePasswordReader.java

... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8d8ce6...01d60c8. Read the comment docs.


Implements ruler for strategy panel

PR related to #
Implements slider (ruler) in strategy panel as shown below.

Need some suggestions on how I can reverse the labels for the y-axis (0-100) instead of (100-0)

Codecov Report

Merging #106 (b83911f) into master (1443d73) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #106   +/-   ##
=========================================
  Coverage     76.78%   76.78%
  Complexity      824      824
=========================================
  Files           121      121
  Lines          2421     2421
  Branches        280      280
=========================================
  Hits           1859     1859
  Misses          474      474
  Partials         88       88

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more


Question about detected help

When I run a report on a module and look at the per function details, the ContainsHelp entry shows false. Which I suppose is technically because there is not comment based help. But there is external help for the module. Are you testing for the existence of comment-based help only or are you taking external help into consideration?

Hi Jeffery,
Yes this is using the GetHelpContent method of FunctionDefinitionAst objects and this method detect only comment-based help.
Another limitation of that way of checking for help, is that it will only detect comment-based help which is inside the body of the function.
For example, it will detect the help in this case :
Function MyFunction () {
<#
.SYNOPSIS
    This cmdlet configures nothing and does it remarkably well.
#>
    Do-Nothing
}

But not in this case :
<#
.SYNOPSIS
    This cmdlet configures nothing and does it remarkably well.
#>
Function MyFunction () {

    Do-Nothing
}

I could use Get-Help but it requires checking if there is a module and loading it, and this would work only for the public functions.
Would you have any suggestions on how to better handle the detection of help ?
Thank you.

I'm not sure about private functions. Since they are only read when editing the module, I would trust the author has enough general comments. I guess I wouldn't worry about those functions. As for public functions, any external help will be an a culture specific folder which you could test for. The file should be called -help.xml.  You could parse the XML looking for matching names.
PS C:\>[xml]$h = get-content c:\scripts\mylmhost\en-us.\Mylmhost-help.xml
PS C:\>$h.helpitems.command.details.name
Get-LmhostsEntry
Remove-LmhostsEntry
Set-LmhostsEntry

You might want to differentiate between comment based help detected or not and external help.

@MathieuBuisson If you go the Get-Help route and want to test private functions you can do something like this:
$module = Get-Module ModuleName
& $module { Get-Help PrivateFunctionName -Category Function }

Also the GetHelpContent method does work for CBH outside the function definition as long as it's included in a parent AST.  I didn't have any issues getting PSCodeHealth to detect CBH in the example you gave.  Is it possible that when you initially tested, it may have been with one of the ASTs that are re-parsed in some of the other private functions?

Hi @SeeminglyScience ,
Well, I haven't actually tested it GetHelpContent against a function which has comment-based help in the parent.
I just assumed it wouldn't find it.
So if it works in this scenario, then it means that this method is more clever than I thought and it is good news.
Thanks.

Hi Jeffery,
Thanks for your help on this (pun not intended).
This has been way more complicated than I expected and it's a pain to unit test, but I managed to make it work.
The function Test-FunctionHelpCoverage now returns $True if there is comment-based help OR if there is a command with the same name listed in an external help file. It uses the technique you showed above to parse the XML content (or MAML).
You can have a look at the commit referenced above this comment to view the code changes.
Could you give it a try with version 0.2.7 of PSCodeHealth available in the PSGallery and let me know if it's OK ?
Thank you.

This looks like it works for me on a few modules that I tested.

@jdhitsolutions ,
That's great.
I'm gonna close this issue then.
Thanks again.


Numba (Python JIT compiler, a little cheating)

a numpy solution not based on matrix multiplication, but adding arrays in a loop. Slower than the matrix solution, on par with the original Python solution: 1.5s processing, 1.5s total
JIT compile the inner loop function with Numba: instantly overtake even scipy solution on processing: 140 ms processing, 650ms total (because of 180ms import and 310ms JIT compile)
sorting to find top5 takes almost half the processing time. replace sorting with one pass over the array to save 55ms (this part probably can be optimized).
other minor optimizations. Final: 60ms processing, 1.2s total (mostly compile)
parallelize the outer loop: 27ms processing, 1.6s total (even longer compile)

Should the parallelized solution be a separate entry in run.sh?
It's not hard to use just one python script for both sequential and parallelized solution, but probably simpler and more consistent with the other solutions to have a different .py file for the parallel version.
Performance measurments: https://github.com/fizmat/related_post_gen_python_timing/blob/main/main.ipynb

create another entry in run.sh for concurrent python, follow the naming of other concurrent solutions

Done. I hope the naming is correct.

Yea, I will merge but I can't put it in the charts as it's not fair to the other langs.


Changed git renamed icon to orange color

Fixes #23939
Changed both dark and light status-renamed.svg to have #CC6633 background color to allow for better distinction as mentioned by issue referenced.

@Kiricon,
Thanks for your contribution.
To ensure that the project team has proper rights to use your work, please complete the Contribution License Agreement at https://cla.microsoft.com.
It will cover your contributions to all Microsoft-managed open source projects.
Thanks,
Microsoft Pull Request Bot

@Kiricon, thanks for signing the contribution license agreement. We will now validate the agreement and then the pull request.
Thanks, Microsoft Pull Request Bot

Sweet, thanks!


Calibrations tutorial

Summary
This PR introduces a calibration tutorial that runs on Armonk.
Details and comments
The PR also has a small fix for timezones in the Calibrations since python cannot compare a timezone aware datetime with one that does not have timezone information.

Are the timestamps saved and displayed both in UTC? Is it possible to save in UTC and display in the user's local time?


Fix URL concatenation in anthropic/api.py

I noticed a small issue with the URL concatenation logic in
https://github.com/anthropics/anthropic-sdk-python/blob/4187c65ae9713b7579fcb15aec43f78ed69b97c4/anthropic/api.py#L71.
The current code uses urllib.parse.urljoin(self.api_url, path) to concatenate URLs, but this results in incorrect URLs when concatenating URLs like
http://localhost:5000/anthropic and /v1/complete, giving
http://localhost:5000/v1/complete instead of the expected
http://localhost:5000/anthropic/v1/complete.
A possible fix could be to directly concatenate the strings, like:
abs_url = "%s%s" % (self.api_base, url)
as implemented in the OpenAI API requestor:
https://github.com/openai/openai-python/blob/db3f3526f16ea518bcd16308508e035017f9259e/openai/api_requestor.py#L467

A possible fix PR: https://github.com/anthropics/anthropic-sdk-python/pull/27


Stable Diffusion Runs with MIGraphX EP

@causten
Previous ref here: https://github.com/ROCmSoftwarePlatform/msft_amd_ai_operators/issues/110
Using this to track progress with stable diffusion runs (prompt->image) via the onnxruntime execution provider using the diffusers library for the v2.0 pipeline.
Current status is as follows:

All individual models have been tested and verified on their own through the MIGraphX driver with @shivadbhavsar's latest changes (4/15/2023)
Currently debugging the EP and using the onnxruntime benchmark script.
Stock results indicate there's in issue with setting the proper sizes during a compile of an MIGraphX program with the CONV operator.

Using MIGRAPHX_TRACE_COMPILE=1 with the onnxruntime EP I'm seeing the following output on the initial text input.
main:@54 = @literal{1, 1} -> int64_type, {2}, {1}
input_ids = @param:input_ids -> int32_type, {1, 1}, {1, 1}

Confirmed from @shivadbhavsar  it appears somehow the EP is getting the wrong parameter size for the input sequence for input_ids which should be 1 64.

Adding this from last weeks notes as well we're supposed to  be using the following to get our driver to run each separate model for the pipeline
bin/driver perf ../vae_decoder_optimized.onnx --input-dim @latent_sample 2 4 64 64 --gpu

bin/driver perf ../vae_encoder_optimized.onnx --input-dim @latent_sample 2 3 512 512 --gpu

bin/driver perf ../text_encode_sd2.onnx --input-dim @input_ids 1 64 --fill1
[06/04 13:49] Austen, Christopher

migraphx-driver perf ./sd-v2-0/unet/model.onnx --input-dim @sample 2 4 64 64 @timestep 1 @encoder_hidden_states 2 64 1024 --fp16

Few updates on this

Looking at the benchmark.py script + diffusers library side, it does appear input_ids is passed through but there's no way to change this cleanly without modifying the library/tests.

Took a look at ROCm EP and they don't run into the issue we're having for our benchmarks as they're adjusting inputs relative to the input data. We should be mirroring this behavior on our end.

Spoke with @pfultz2 about this and the notion seems that we need to add another condition for variable tensor dimensions in our EP to avoid an initial compile with default input_ids which may cause erroneous states. This is seen when we get passed the "Default" input size, but then fail on compile for Conv Channel checks.

Removing the entire section that was causing issue for now + added probing and it appears we are getting runs now but we're slightly slower than ROCm

Required fixes for missing ops has been merged into ORT: https://github.com/microsoft/onnxruntime/pull/15463
Coming up with another PR with changes for the EP and will link to this ticket.
Currently able to get runs by disabling our precompile to set input shapes, as @pfultz2 suggested, if we can't modify
the library/benchmark input.
Inference took 3.688 seconds {'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 3.7156848907470703, 'median_latency': 3.7084567546844482, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': 'sd-v2-0-migx/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

This is with logging added with the EP and just taking the tip of MIGX develop.
Retesting with removing logging + working off the perf_4_19_23 branch

Run with
export MIGRAPHX_DISABLE_CK_GEMM=1
export MIGRAPHX_DISABLE_CK_GEMM_FUSION=1
export MIGRAPHX_DISABLE_CK_GEMM_SOFTMAX_GEMM=1

python optimize_pipeline.py  -i ./sd-v2 -o ./sd-v2-0-migx --float16 --disable_attention  --disable_skip_layer_norm
--disable_embed_layer_norm --disable_bias_skip_layer_norm  --disable_nhwc_conv   --disable_group_norm

Model loading took 20.26311230659485 seconds
Inference took 3.709 seconds
Inference took 3.609 seconds
Inference took 3.602 seconds
Inference took 3.596 seconds
Inference took 3.598 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 3.622612237930298, 'median_latency': 3.6021175384521484, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': 'sd-v2-0-migx/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Ted was using the perf_4_19_23 branch.  Slower means the optimize_pipeline.py  is finding more optimizations

@TedThemistokleous is there a place to check if there are operators not consumed by MIGX?  Is there a way to dump the timing between the 3 models?  Can you dump the graph the way migx optimizes it?  Would be good to see if it is the same as what we get with just our driver

I can dump the runs by using migraphx flags for eval, thats how I found the issue with input sizes. I can also turn on logging for the benchmark script again and dump this to a log

Was trying different optimizations right now but I seem to be getting no output at all when I perform new benchmarks with the reoptimzied model

Spent the afternoon playing with turning off the optimizer by adding a no_optimize flag into the optimizer script itself to leverage the existing conversion to fp16. There seems to be some other things happening in the background thats assigning fusions to the specific nodes.
As a sanity check, I tried a few runs with stock step size and step size of 1 on the unoptimized model between ROCm and MIGraphX and I did see that MIGraphX is 2x faster.
Namespace(batch_count=5, batch_size=1, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0/', provider='migraphx', steps=1, use_xformers=False, version='1.5', width=512)
GPU memory used before loading models: -1
Model loading took 20.00900173187256 seconds
Inference took 0.793 seconds
Inference took 0.798 seconds
Inference took 0.789 seconds
Inference took 0.795 seconds
Inference took 0.820 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 1, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 0.7991061687469483, 'median_latency': 0.7951765060424805, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'runwayml/stable-diffusion-v1-5', 'directory': './sd-v2-0/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

vs
Inference took 1.608 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 1, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 1.660592269897461, 'median_latency': 1.6076416969299316, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0', 'provider': 'ROCMExecutionProvider', 'disable_safety_checker': True}

fp32 bench
Inference took 60.296 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 59.526450634002686, 'median_latency': 59.64444899559021, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0', 'provider': 'ROCMExecutionProvider', 'disable_safety_checker': True}

Inference took 5.217 seconds
Inference took 5.251 seconds
Inference took 5.235 seconds
Inference took 5.237 seconds
Inference took 5.217 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 5.231530046463012, 'median_latency': 5.2354490756988525, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Looks like there was section in the EP that was checking input sizes and throwing all our resizes on the CPU which was what hte bottleneck was
Output using Batchsize 1
Namespace(batch_count=5, batch_size=1, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512) GPU memory used before loading models: -1
Model loading took 20.58418107032776 seconds
Inference took 2.183 seconds
Inference took 2.182 seconds
Inference took 2.185 seconds
Inference took 2.183 seconds
Inference took 2.181 seconds
 {'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 2.182928276062012, 'median_latency': 2.1831724643707275, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Output using Batchsize 4
Namespace(batch_count=5, batch_size=4, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512) GPU memory used before loading models: -1
Model loading took 21.238120555877686 seconds
Inference took 6.912 seconds
Inference took 6.922 seconds
Inference took 6.901 seconds
Inference took 6.917 seconds
Inference took 6.918 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 6.913937950134278, 'median_latency': 6.917336463928223, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Output using Batchsize 1 for OEM benchmark
root@c967699ee7ef:/workspace/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/ -r migraphx -v 2.0 -b 1 & ROCR_VISIBLE_DEVICES="5" python3 benchmark.py -p ./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/ -r migraphx -v 2.0 -b 1
[1] 771250
Namespace(batch_count=5, batch_size=1, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Namespace(batch_count=5, batch_size=1, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 20.744670629501343 seconds
Model loading took 20.930532932281494 seconds
Inference took 2.197 seconds
Inference took 2.202 seconds
Inference took 2.201 seconds
Inference took 2.200 seconds
Inference took 2.201 seconds
Inference took 2.206 seconds
Inference took 2.201 seconds
Inference took 2.203 seconds
Inference took 2.202 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 2.200548839569092, 'median_latency': 2.2008516788482666, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
Inference took 2.201 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 2.2023781299591065, 'median_latency': 2.2020533084869385, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
[1]+  Done                    ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/ -r migraphx -v 2.0 -b 1

Output using Batchsize 4 for OEM
root@c967699ee7ef:/workspace/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/ -r migraphx -v 2.0 -b 4 & ROCR_VISIBLE_DEVICES="5" python3 benchmark.py -p ./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/ -r migraphx -v 2.0 -b 4
[1] 672646
Namespace(batch_count=5, batch_size=4, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Namespace(batch_count=5, batch_size=4, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 21.340993642807007 seconds
Model loading took 21.66949462890625 seconds
Inference took 13.487 seconds
Inference took 14.288 seconds
Inference took 14.618 seconds
Inference took 14.586 seconds
Inference took 15.049 seconds
Inference took 15.116 seconds
Inference took 14.483 seconds
Inference took 14.003 seconds
Inference took 14.333 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 14.393894290924072, 'median_latency': 14.483012437820435, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
Inference took 14.298 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 14.458590269088745, 'median_latency': 14.298325777053833, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
[1]+  Done                    ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/ -r migraphx -v 2.0 -b 4

Here's the driver output after I just do an MIGraphX run with the model used for Unet to get the above results. Not sure if there's something we can also improve with say softmax or layernorm?  @pfultz2
gpu::gemm: 19.0171ms / 196 = 0.0970262ms, 37%
gpu::convolution: 14.91ms / 66 = 0.225909ms, 29%
gpu::code_object::softmax_kernel: 4.17143ms / 32 = 0.130357ms, 8%
gpu::code_object::layernorm_kernel: 2.74906ms / 61 = 0.0450666ms, 6%
gpu::code_object::contiguous_kernel: 2.67727ms / 100 = 0.0267727ms, 6%
gpu::code_object::add_layernorm_kernel: 1.49249ms / 48 = 0.0310935ms, 3%
gpu::code_object::mul_add_sigmoid_mul_kernel: 1.36453ms / 45 = 0.0303229ms, 3%
gpu::code_object::add_kernel: 1.32142ms / 24 = 0.0550591ms, 3%
gpu::code_object::add_add_kernel: 1.19074ms / 39 = 0.0305319ms, 3%
gpu::code_object::mul_neg_exp_add_div_mul_kernel: 0.7844ms / 16 = 0.049025ms, 2%
load: 0.598918ms / 722 = 0.000829527ms, 2%
gpu::code_object::concat_add_kernel: 0.521108ms / 8 = 0.0651385ms, 1%
gpu::code_object::concat_kernel: 0.477438ms / 16 = 0.0298399ms, 1%
hip::hip_copy_literal: 0.427272ms / 456 = 0.000937ms, 1%
reshape: 0.343927ms / 374 = 0.000919592ms, 1%
multibroadcast: 0.211719ms / 184 = 0.00115065ms, 1%
slice: 0.188216ms / 136 = 0.00138394ms, 1%
transpose: 0.150454ms / 160 = 0.000940335ms, 1%
broadcast: 0.0598209ms / 52 = 0.0011504ms, 1%
gpu::code_object::add_sigmoid_mul_kernel: 0.0449055ms / 2 = 0.0224528ms, 1%
gpu::code_object::concat_add_add_kernel: 0.0233836ms / 1 = 0.0233836ms, 1%
gpu::code_object::sin_kernel: 0.0163115ms / 1 = 0.0163115ms, 1%
gpu::code_object::cos_kernel: 0.0159051ms / 1 = 0.0159051ms, 1%
gpu::code_object::mul_kernel: 0.0158072ms / 1 = 0.0158072ms, 1%
@param: 0.0037551ms / 4 = 0.000938775ms, 1%
check_context::migraphx::gpu::context : 0.00167932ms / 1 = 0.00167932ms, 1%
unsqueeze: 0.00158086ms / 1 = 0.00158086ms, 1%
hip::hip_allocate_memory: 0.00122494ms / 1 = 0.00122494ms, 1%

Batch size: 1
Rate: 26.5349/sec
Total time: 37.6862ms
Total instructions time: 52.7819ms
Overhead time: 0.886083ms, -15.0958ms
Overhead: 2%, -40%

How much of a speedup do we expect?  This is for Batch size 4.
Namespace(batch_count=5, batch_size=4, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 20.901840925216675 seconds
Inference took 6.441 seconds
Inference took 6.391 seconds
Inference took 6.385 seconds
Inference took 6.400 seconds
Inference took 6.387 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 6.400722599029541, 'median_latency': 6.391122579574585, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sd-v2-0-fp16-no-batt-ln-bias-gn-sh-packed/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Going to try CK

Sorted out CK and got my own tunings running for batch size 1 and 4 (rename this to .json in linux) Testing this now.
gfx90a_ck.txt

GCD benchmark with my tunings
BS=1
Inference took 2.058 seconds
Inference took 2.059 seconds
Inference took 2.058 seconds
Inference took 2.057 seconds
Inference took 2.058 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 2.0580252170562745, 'median_latency': 2.0581021308898926, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

This looks like about a ~10% bump with CK (2.2s -> 2.0s)
BS =4
Inference took 6.488 seconds
Inference took 6.424 seconds
Inference took 6.505 seconds
Inference took 6.439 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 6.456205415725708, 'median_latency': 6.438835144042969, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Don't see much of a difference for bs 4. Will try OAM now

CK with OAM Bench
Bs =1
ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b1 & ROCR_VISIBLE_DEVICES="5" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b1

Inference took 2.079 seconds
Inference took 2.088 seconds
Inference took 2.084 seconds
Inference took 2.081 seconds
Inference took 2.084 seconds
Inference took 2.085 seconds
Inference took 2.081 seconds
Inference took 2.087 seconds
Inference took 2.082 seconds
Inference took 2.082 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 2.082117223739624, 'median_latency': 2.0822181701660156, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 1, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 2.08473801612854, 'median_latency': 2.0853352546691895, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

BS = 1
ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b4 & ROCR_VISIBLE_DEVICES="5" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b4

Inference took 13.200 seconds
Inference took 13.273 seconds
Inference took 14.196 seconds
Inference took 14.104 seconds
Inference took 15.326 seconds
Inference took 15.238 seconds
Inference took 13.491 seconds
Inference took 13.579 seconds
Inference took 14.957 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 14.233848047256469, 'median_latency': 14.19596552848816, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
Inference took 14.955 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 4, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 14.229776954650879, 'median_latency': 14.104409456253052, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

Going to repeat CK tests using the tunings from @causten that were also done for bert and batch size 1.

Had an issue using the tunings from Chris so I've moved to seeing where we fail on largest batch size wihtout CK
BS = 8
root@27ec8e19d6fb:/code/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b 8
Namespace(batch_count=5, batch_size=8, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sdv2-0-migx-noops-fp16/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 19.83580708503723 seconds
Inference took 13.333 seconds
Inference took 13.251 seconds
Inference took 13.295 seconds
Inference took 13.252 seconds
Inference took 13.290 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 8, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 13.284286975860596, 'median_latency': 13.290247201919556, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

BS = 16
root@27ec8e19d6fb:/code/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b 16
Namespace(batch_count=5, batch_size=16, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sdv2-0-migx-noops-fp16/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 19.897315740585327 seconds
Inference took 25.812 seconds
Inference took 25.793 seconds
Inference took 25.807 seconds
Inference took 25.774 seconds
Inference took 25.788 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 16, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 25.79488558769226, 'median_latency': 25.793336629867554, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

BS = 32 seems to be our limit
root@27ec8e19d6fb:/code/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b 32
Namespace(batch_count=5, batch_size=32, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sdv2-0-migx-noops-fp16/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 19.819168090820312 seconds
MIGraphX Error: /code/migraphx/src/src/targets/gpu/hip.cpp:109: allocate_gpu: Memory not available to allocate buffer: 251658240
2023-04-28 18:15:04.843686691 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_torch_jit_15298953127787892160_0 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_torch_jit_15298953127787892160_0_0' Status Message: Failed to call function
[ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MGXKernel_graph_torch_jit_15298953127787892160_0 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_torch_jit_15298953127787892160_0_0' Status Message: Failed to call function
root@27ec8e19d6fb:/code/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion#

Test with Batch size = 3 to meet A100 bench
root@27ec8e19d6fb:/code/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b 3
Namespace(batch_count=5, batch_size=3, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sdv2-0-migx-noops-fp16/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
Model loading took 20.027427434921265 seconds
Inference took 5.420 seconds
Inference took 5.428 seconds
Inference took 5.417 seconds
Inference took 5.421 seconds
Inference took 5.415 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 3, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 5.420321655273438, 'median_latency': 5.420013904571533, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}

BS = 3 OAM bench. Seems lower than BS = 4
root@27ec8e19d6fb:/code/onnxruntime/onnxruntime/python/tools/transformers/models/stable_diffusion# ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b3 & ROCR_VISIBLE_DEVICES="5" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b3
[1] 2578897
Namespace(batch_count=5, batch_size=3, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sdv2-0-migx-noops-fp16/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
Namespace(batch_count=5, batch_size=3, enable_safety_checker=False, enable_torch_compile=False, engine='onnxruntime', height=512, num_prompts=1, pipeline='./sdv2-0-migx-noops-fp16/', provider='migraphx', steps=50, use_xformers=False, version='2.0', width=512)
GPU memory used before loading models: -1
GPU memory used before loading models: -1
Model loading took 20.101964712142944 seconds
Model loading took 20.239580631256104 seconds
Inference took 13.943 seconds
Inference took 13.888 seconds
Inference took 13.826 seconds
Inference took 13.797 seconds
Inference took 13.109 seconds
Inference took 13.167 seconds
Inference took 12.977 seconds
Inference took 12.943 seconds
Inference took 12.909 seconds
Inference took 13.227 seconds
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 3, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 13.357697248458862, 'median_latency': 13.1669020652771, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
{'engine': 'onnxruntime', 'version': '1.15.0', 'height': 512, 'width': 512, 'steps': 50, 'batch_size': 3, 'batch_count': 5, 'num_prompts': 1, 'average_latency': 13.399609804153442, 'median_latency': 13.226958751678467, 'first_run_memory_MB': -1, 'second_run_memory_MB': -1, 'model_name': 'stabilityai/stable-diffusion-2', 'directory': './sdv2-0-migx-noops-fp16/', 'provider': 'MIGraphXExecutionProvider', 'disable_safety_checker': True}
[1]+  Done                    ROCR_VISIBLE_DEVICES="4" python3 benchmark.py -p ./sdv2-0-migx-noops-fp16/ -r migraphx -v 2.0 -b3

Running last set of CK tests, will post benchmark script and tunings to this ticket as well as final log.
Changes to onnxruntime PR found here and will be finalized once I retest cleanup changes
https://github.com/microsoft/onnxruntime/pull/15755

gfx90a_ck_SD_1_2_3_4_8_16.txt
benchmark_runs_ck.log
Here's the output from the test runs
It appears we've hit a limit with OAM benchmarks when using batch size = 16

The script used doesn't capture the second OAM run but the values are within the same parameters.
PR changes have been posted and waiting further comments/runs from ORT team.


Want to Add Weather App

Feature Type:

Want to Add a weather app using Javascript.
It will be a basic one.

We already have a weather app


"next branch"  synchronize to handle event

Per my understanding mesos can not guarantee the time sequence of event delivery to framework,
next branch
https://github.com/mesos/mesos-go/blob/next/extras/scheduler/controller/controller.go#L103
Will the synchronize to handle event lead to decread of throughput?
Is it better to change it into goroutine?

AFAIK mesos tries (but doesn't guarantee, due to libprocess reasons) to
deliver status updates, for some task, in the order they were processed by
the master. There's at least one Mesos JIRA tracking/discussing that issue.
I deliberately chose to NOT use multiple goroutines here for dispatching
event handling. This allows a framework scheduler to implement a simpler
set of handlers and to worry less about the state of the system changing
under it. While it's possible that throughput could be improved with
multiple goroutines I strongly suspect that other parts of the cluster
would become a bottleneck first. If some framework-specific event handler
has to perform any kind of potentially blocking work then it probably
makes sense to run that work in a goroutine (perhaps via a job queue) to
avoid slowing down the controller/event-loop machinery.
On Mon, Jun 20, 2016 at 10:51 AM, wangwei<EMAIL_ADDRESS>wrote:

Per my understanding mesos can not guarantee the time sequence of event
delivery to framework,
next branch
https://github.com/mesos/mesos-go/blob/next/extras/scheduler/controller/controller.go#L103
Will the synchronize to handle event lead to decread of throughput?
Is it better to change it into goroutine?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/mesos/mesos-go/issues/251, or mute the thread
https://github.com/notifications/unsubscribe/ACPVLAs5qViBRcI7QLAuuFo_hin3RVedks5qNqjugaJpZM4I5xzn
.

I did a test. In the HandleEvent, after the receipt of offer, a lot of tasks which are docker containers and could automatically exit after <10s running were launched by using goroutine, and block the HandleEvent by using time.sleep for a relarively long time (such as 100 seconds). The tasks which had been excited in this 100s will release the offer, but some offer event was missing when calling framework via HandleEvent again. I might run into trouble when using bounded offer hoarding to design schedule. I should set a --offer_timeout in the master. Is it the long time of sleep in the HandleEvent leading to the missing? Or something else would lead to the missing.

If you could share some code (or pseudo-code) that would really help me understand more of the problem you're running into. If you're sleeping in an event handler then you're basically blocking further communication from mesos during that time; I strongly suggest that you consider an alternative implementation strategy.

I worked on this case and I will try to make it clear.
I wrote a simple handler for Event_OFFERS
scheduler.Event_OFFERS: func(e *scheduler.Event) error {
				go driver.ResourceOffers(e.GetOffers().GetOffers()) // Just decline it.
				if isfirst { //this is a variable with value 'true' to make the handler sleep 30s only for the first event.
					time.Sleep(time.Second * 30)
					isfirst = false
				}
				return nil
},

If there are two or more offer events come in a short time. Some offers will be probably dropped and will be never declined. These offers will stay on the Outstanding Offers list of the mesos master and waiting for recind.
Then we tried on changing the buffer size. We make a little change on NewFrameReader
func NewFrameReader(r io.Reader) framing.Reader {
	// br, ok := r.(*bufio.Reader)
	// if !ok {
	// 	br = bufio.NewReader(r)
	// }
	br := bufio.NewReaderSize(r, 1024*1024*1024)
	return &reader{r: br}
}

The bigger buffer we have, the less offers we missed.
I found this issue when the scheduler in high load capacity. Many tasks finished and many tasks launched in seconds. Offer events missed even though I handle it with goroutines. This issue would cut off the offers controlled by the scheduler more and more in every recind cycle.

@jdef
it might be an extreme example (sleep 30)
but I have made a test using the above-mentioned method. after the revision of the buffsize in the https://github.com/mesos/mesos-go/blob/next/recordio/reader.go#L22
the loss of event message indeed gets improved is flaky
will it block the write operation in the server-side when the event processing get slowdown(eg:sleep n mock...)?

your implementation of go driver.ResourceOffers(e.GetOffers().GetOffers()) does not comply with the contract of the scheduler callbacks: in your impl it's possible that multiple driver callback funcs may be invoked concurrently. this should not happen. in fact the impl in /master not only serializes execution of callbacks but also attempts to preserve the order in which the events arrived from the master (the executor driver impl is lazier w/ respect to ordering but still guarantees serial callback execution).
it seems like your test case attempts to prove that when a scheduler doesn't respond for some period of time that some offers may never be received? the current /next implementation implements back-pressure: the stream of events sent by the master is only processed as fast as the handler impl can take them. so if the handler blocks somewhere (for say 30s) then the event stream might be in some transient state. for example, the event stream buffer could contain offers that are already expired.
there could be some other blocking condition, like a network partition. i'm working on a heartbeat monitor for this. in either case the scheduler must be ready to deal with network / event-stream failures. this is a distributed system after all and failures are pretty much guaranteed at some point.
the key here is to NOT block in the event stream handlers. process events as quickly as possible in the handlers and if there's any blocking work to do then it should be queued and processed outside of the handler loop.
does that make sense, or am I missing something obvious?

considering this resolved. please re-open if needed


[s3] New IAM user created without user's knowledge

Steps

Go to the settings page of a project for which you have write access.
Select the "Amazon Simple Storage Service" item and click submit.
Enter in AWS Secret Key and Access Key to link your S3 account.
Refresh page and confirm that your account as been linked.

Expected
The user should expect that the OSF now has access to their S3 account using the credentials given. When deleting this key on the S3 site, OSF should no longer have access to their account.
Actual
An IAM user is created on the user's S3 account without their knowledge:

Therefore, if a user wants to deauthorize the OSF from accessing their Amazon account and wants to complete this action from the S3 console, they have to delete an IAM user they did not know was created. Deleting the secret/access keys given to the OSF has no effect. The user should be warned about these actions on linking their account.

I propose we leave this as-is. The solution in place now considered this potential confusion, and we decided that having a discrete user for the OSF outweighed the negatives.
In light of that, I'm going to go ahead and close this. If anyone feels it has not been adequately addressed, by all means re-open it.