Version bump crashes

bump-my-version version: 0.15.1
Python version: 3.11
Operating System: macOS Sonoma

Description
Version bump crashes right before creating a commit and throws a KeyError.
Based on quick check - this issue started to appear since version 0.14.0.
What I Did
bump-my-version bump patch
<click stacktrace>
  File "/.../.virtualenv/.../lib/python3.11/site-packages/bumpversion/cli.py", line 310, in bump
    do_bump(version_part, new_version, config, found_config_file, dry_run)
  File "/.../.virtualenv/.../lib/python3.11/site-packages/bumpversion/bump.py", line 100, in do_bump
    update_config_file(config_file, config, version, next_version, ctx, dry_run)
  File "/.../.virtualenv/.../lib/python3.11/site-packages/bumpversion/config/files.py", line 145, in update_config_file
    updater.update_file(current_version, new_version, context, dry_run)
  File "/.../.virtualenv/.../lib/python3.11/site-packages/bumpversion/files.py", line 307, in update_file
    self._update_toml_file(search_for, raw_search_pattern, replace_with, dry_run)
  File "/.../.virtualenv/.../lib/python3.11/site-packages/bumpversion/files.py", line 322, in _update_toml_file
    raise ValueError(
ValueError: Key 'tool.bumpversion.current_version' in /.../pyproject.toml does not contain the correct contents: 0.1.26
make: *** [bump_patch] Error 1

pyproject.toml configuration (before bump):
[tool.bumpversion]
current_version = "0.1.26"
allow_dirty = true
commit = true

[[tool.bumpversion.files]]
filename = "pyproject.toml"
search = "version = \"{current_version}\""
replace = "version = \"{new_version}\""

Thanks for reporting this so quickly. I'll focus on it immediately

@rgryta Just a quick update. As I was looking at what you were doing, I believe it is related to changes on how .toml files are processed. (and obviously you found some unintended consequences.)
I introduced key_path into the file config to allow for more intelligent replacement in structured data files.
While I update the documentation and fix incompatibility with old search/replace methods in toml files... Try this:
[tool.bumpversion]
current_version = "0.1.26"
allow_dirty = true
commit = true

[[tool.bumpversion.files]]
filename = "pyproject.toml"
key_path = "<path.to>.version"

An example for standard pyproject.toml version: key_path = "project.version"
Or for poetry: key_path = "tool.poetry.version"

@rgryta Just a quick update. As I was looking at what you were doing, I believe it is related to changes on how .toml files are processed. (and obviously you found some unintended consequences.)
I introduced key_path into the file config to allow for more intelligent replacement in structured data files.
While I update the documentation and fix incompatibility with old search/replace methods in toml files... Try this:
[tool.bumpversion]
current_version = "0.1.26"
allow_dirty = true
commit = true

[[tool.bumpversion.files]]
filename = "pyproject.toml"
key_path = "<path.to>.version"

An example for standard pyproject.toml version: key_path = "project.version"
Or for poetry: key_path = "tool.poetry.version"

I was having the same issue as @rgryta on v0.15.3 (at least in dry run mode, haven't tried to use it for real yet). I tried key_path but it does not seem to prevent the search-and-replace of pyproject.toml from modifying tool.bumpversion.current_version (when the intention is to update tool.poetry.version). Here is what I tried:
[[tool.bumpversion.files]]
filename = "pyproject.toml"
key_path = "tool.poetry.version"

@proc-2 Can you verify that you are using 0.15.3?
I'm asking because I have a specific test for this issue.
The real issue is that the generic search for version = "{current_version" matches both the line starting with version and bumpversion's own current_version.  Prior to the fix, when bump-my-version tries to update its own configuration it detects the value is different than expected, and explodes.
If you are really using 0.15.3, I'll need more information to be able to reproduce (and re-open) this.


the big revamp PR!

Are you a current Fivetran customer?

What change(s) does this PR introduce?

Did you update the CHANGELOG?

[ ] Yes

Does this PR introduce a breaking change?

[ ] Yes (please provide breaking change details below.)
[ ] No  (please provide an explanation as to how the change is non-breaking below.)

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

[ ] Yes

Is this PR in response to a previously created Bug or Feature Request

[ ] Yes, Issue/Feature [link bug/feature number here]
[ ] No

How did you test the PR changes?

[ ] Buildkite
[ ] Local (please provide additional testing details below)

Select which warehouse(s) were used to test the PR

[ ] BigQuery
[ ] Redshift
[ ] Snowflake
[ ] Postgres
[ ] Databricks
[ ] Other (provide details below)

Provide an emoji that best describes your current mood

:dancer:
Feedback
We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your feedback on our existing dbt packages or what you'd like to see next.

@fivetran-jamie One last item. For posterity, would you be able to update the description of the PR. Just to highlight the other PRs that were merged into here.


Added all base materials from gatherable resources

You can happily delete my "comments" to sort it, was just for my own overview of all the things

There are duplicates (at a minimum, light birch) and errors (arcaneleaf and wolfsbane). Fix those and take out the tradeskill name comments and this will be a great addition.
For finding duplicates in Notepad++, you might try the Function List plugin (https://notepad-plus-plus.org/features/function-list.html). There's also PyLint (https://www.pylint.org/) which will do all kinds of code analysis for you in addition to finding duplicate functions.


line length warnings, and general compatibility with linter used with hhvm

I had a PR to HHVM rejected since several lines after a refactor were > 80 characters long
https://reviews.facebook.net/D21579
I built the OSS version of flint and ran it over the changed files in the PR, and flint was perfectly happy, with no complaints about line length.  Indeed, nothing in flint --help says there's a knob to control the line length threshold.
How do I run flint a-priori on my workstation prior to making a PR using OSS tools so that it finds the same set of errors that are discovered for the tools feeding phabricator for the HHVM project?

Thank you for reporting this issue and appreciate your patience. We've  notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.


New Azure Gov regions

Need to update this documentation to reflect the new Azure Government regions: https://azure.microsoft.com/en-us/blog/new-government-datacenter-regions-available-in-arizona-and-texas/

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a0acd798-2e5a-2a7e-bb14-7424576a8e38
Version Independent ID: 841f2ce1-b101-0fbf-e33d-4e56a6244ed9
Content: Locations and connectivity providers: Azure ExpressRoute
Content Source: articles/expressroute/expressroute-locations-providers.md
Service: expressroute
GitHub Login: @cherylmc
Microsoft Alias: jaredro

@cap273 Thanks for the feedback! I have assigned the issue to the content author to evaluate and update as appropriate.

@jaredr80 - can you please take a look?

Respectfully following up.

I just followed up on this. It's been updated. Thanks! #please-close


Implemented GEOSRelatePattern as BaseGeometry.relate_pattern

This commit implements GEOSRelatePattern as object.relate_pattern, e.g.:
>>> g1 = Polygon([(0, 0), (0, 1), (3, 1), (3, 0), (0, 0)])
>>> g2 = Polygon([(1, -1), (1, 2), (2, 2), (2, -1), (1, -1)])
>>> g1.relate_pattern(g2, 'T********')
True

Needs documentation, although not too much more than is already provided for object.relate. http://toblerity.org/shapely/manual.html#de-9im-relationships

@snorfalorpagus looks good! Do you want to add something to the manual in this PR or in another?

I'll bundle the changes to the manual into this PR.

@snorfalorpagus can you make a new PR for maint-1.5 as in #295? I owe you several :beers:


Update master from develop

:white_check_mark: Build https://ci.appveyor.com/project/oxidemod/oxide/build/3066-oggoajdn|#3066-oggoajdn (https://github.com/OxideMod/Oxide/commit/05a562bf35|05a562bf35) by @lukespragg) completed


[Feature] Hive Source/Sink support multiple table

Purpose of this pull request

Use Factory API to create Hive Source/Sink
Hive Source/Sink support multiple table

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

Hi, @ruanwenjun  Is there any new progress in this PR?

Hi, @ruanwenjun Is there any new progress in this PR?

I have resolve the conflicts.

Please add test case

We can add test cases later
cc @EricJoy2048 @Hisoka-X

Could you please offer some snapshots to verify it worked? For new hive sink/source has no e2e cases, so I think offer some screenshots would be better.
I remember before, the community was formulating relevant regulations. If the merged code can write e2e, e2e should be written. If it cannot be written, relevant information should be provided to explain why it cannot be written. At the same time, some evidence should be provided to prove that this PR is effective and can be used in production.
cc @EricJoy2048 @hailin0 @Hisoka-X

I have successfully applied this solution and the configuration file is as follows
After a period of operation, it runs stably The table data information is as follows table one  table two  I have also tried PG and Mongodb as data sources and have been successful

Thanks


RTOS Nonpreemptive Static Scheduling

As a reminder we will also need to update the repository config.json file when we merge this

As a reminder we will also need to update the repository config.json file when we merge this

What difficulty level are you thinking we should put this exercise at?

As a reminder we will also need to update the repository config.json file when we merge this

What difficulty level are you thinking we should put this exercise at?

I think this one could potentially be considered a hard exercise


test: cleanup

Two minor things:

define testFn.message early, otherwise it is undefined at the moment it is checked and then useless,
use ifEnvSupports for Promise tests

Landed as b7e5b3c4b94a08d5945cc0a7589fa1e15b8613ef.


[Feature Request] Have option to return trace as part of out put of a PromptFlow when deployed as endpoint

Is your feature request related to a problem? Please describe.
I want instant feedback after running a deployed prompt so I can put it in my database.
I currently need to keep polling for the azure insights to find the data I want so I need a background job to update the table.
Describe the solution you'd like
Make it possible to output the trace as part of the endpoint response. This way when I get the results I also have the telemetry so I can apply to tables along with results.
Describe alternatives you've considered
I have created a background worker to keep polling the azure insights for the prompt event which adds unnecessary complexity. It would be much easier if I can get the data immediately.
Additional context
This is a deployed endpoint.

Hi @adampolak-vertex , thanks for reaching out!
Based on my understanding of trace and open telemetry, when the endpoint returns response, the corresponding trace/span is not close yet, so it shall not be carried to the endpoint response. Besides, as deployed endpoint does not expose any other API to read, so it can be difficult to make you access the trace data.
For your scenario, is it possible for you to host a separate server somewhere else, and it is accessible for you? If so, I think you can host a prompt flow service there, and pointing your deployed endpoint to export traces to that compute; then you are able to access the trace data freely - feel free to share your thought on this solution, and I can provide more information if it works for you; otherwise, I can sync with my team and brainstorm some other possible solution(s).

Hi @adampolak-vertex, instead of sending the trace back as part of the response, is it possible that you implement your own trace exporter which directly send the trace to your database?  This is the suggested solution for exporting OTel traces.

Hi @adampolak-vertex , thanks for reaching out!
Based on my understanding of trace and open telemetry, when the endpoint returns response, the corresponding trace/span is not close yet, so it shall not be carried to the endpoint response. Besides, as deployed endpoint does not expose any other API to read, so it can be difficult to make you access the trace data.
For your scenario, is it possible for you to host a separate server somewhere else, and it is accessible for you? If so, I think you can host a prompt flow service there, and pointing your deployed endpoint to export traces to that compute; then you are able to access the trace data freely - feel free to share your thought on this solution, and I can provide more information if it works for you; otherwise, I can sync with my team and brainstorm some other possible solution(s).

Thank you for your quick response.
I am currently using the Azure UI to deploy as that is a direction our team wants to go for ease of use to other stakeholders.
How can I point a trace to a different server from my endpoint?
In returning it with the api I did not mean a new route, but rather another key it could return.
So it would be: output.answers and output.trace, so a single response would return both as different keys.

Hi @adampolak-vertex, instead of sending the trace back as part of the response, is it possible that you implement your own trace exporter which directly send the trace to your database? This is the suggested solution for exporting OTel traces.

How would I do this using the Azure UI? Is that possible or does it need to not be deployed through azure portal or does that not make a difference?
I would be happy to add a trace to the beginning of my promptflow and then export it out at the end along with the output, but how would that work?

To point traces to different server, you can use OTel environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to do so, PF will honor this setting and export traces to that endpoint - this requires you to host a trace server on that place.
For returning in API, I think the answer is still no, as I mentioned, when API returns the response, the trace is still unavailable at that time.
BTW, could you please describe more on the database/table where you store the traces? I agree with Xiaopeng's suggestion that you can write your own exporter to export traces to your database, as it's the recommended way from OTel - this is something written in your code.

To point traces to different server, you can use OTel environment variable OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to do so, PF will honor this setting and export traces to that endpoint - this requires you to host a trace server on that place.
For returning in API, I think the answer is still no, as I mentioned, when API returns the response, the trace is still unavailable at that time.
BTW, could you please describe more on the database/table where you store the traces? I agree with Xiaopeng's suggestion that you can write your own exporter to export traces to your database, as it's the recommended way from OTel - this is something written in your code.

How would I set the environment variable for PromptFlow if I deployed as an endpoint using the Azure Portal UI?
I know OpenAI responds with tokens used per call, so at the least could the tokens be returned as part of the AI response?

I don't see place to set environment variable in portal UI, not sure it is supported from UI; however, is it possible to set it in your flow? I understand those code are controlable from your side?

Meanwhile, token should be available, but it's different from tracing. Providing token usage in endpoint response should be a separate topic from prompt flow; tagging Xiaopeng @wxpjimmy to add some comments on this.

Meanwhile, token should be available, but it's different from tracing. Providing token usage in endpoint response should be a separate topic from prompt flow; tagging Xiaopeng @wxpjimmy to add some comments on this.

That would be a good first step. Because

time spent (should be instant)
prompt tokens (should be instant)
inputs and outputs of every module (should be instant)

None of these are post-processing values so it would be great to have an option "include logs in response.logs and the result is in response.output.

If you want to get token, you can write your own llm tool, in that tool you can parse the response and return both output message and token info as output, PF doesn't have any plan to provide these info in the response.

@adampolak-vertex what you need is still the trace data, please follow the suggested practice to define your own exporter to collect those info, PF doesn't have plan to provide that info in the response from infra perspective.  cc @zhengfeiwang


feedback

Had a nice introduction to the concept of loss-aversion and what their paper
was studying (it was one of the clearest of the introductions on the
loss-aversion study).
Only group that really introduced the models they were intending to use. They've
clearly done a good job reading and understanding their paper.
Preliminary results/preprocessing results were noticably absent; a lot of
focus on information from the paper.  Seemed a bit behind in working with the
fMRI data.
Overall, seemed to have a very nice conceptual understanding of your study,
but really need to get working with the data. Your presented goal was
reasonable and feasible (propose to repeat the analysis in the paper to
validate the models and regression techniques used) but it seemed like very
little concrete work has materialized toward that end.
You need to make sure you are writing valid code.  For example, you can't use
np.something, if np doesn't exist.

closing, thanks for feedback


docs: Documentation bug re: $ionicPlatform.currentPlatfo

Type: docs
Platform: all
Re: here:
http://ionicframework.com/docs/api/utility/ionic.Platform/
https://github.com/driftyco/ionic/blob/master/js/utils/platform.js#L15
What are the possible values?  ios, android, windowsphone, blackberry?, tizen?, etc?
Thanks!
Mike

Hmm, I'm not seeing $ionicPlatform.currentPlatform() anywhere in the docs.
angular.module('PlatformApp', ['ionic'])
.controller('PlatformCtrl', function($scope) {

  ionic.Platform.ready(function(){
    // will execute when device is ready, or immediately if the device is already ready.
  });

  var deviceInformation = ionic.Platform.device();

  var isWebView = ionic.Platform.isWebView();
  var isIPad = ionic.Platform.isIPad();
  var isIOS = ionic.Platform.isIOS();
  var isAndroid = ionic.Platform.isAndroid();
  var isWindowsPhone = ionic.Platform.isWindowsPhone();

  var currentPlatform = ionic.Platform.platform();
  var currentPlatformVersion = ionic.Platform.version();

  ionic.Platform.exitApp(); // stops the app
});

This will actually call the cordova device plugin so it will use that to get the platform. So theoretically it would return the right platform for every platform cordova supports, even if ionic doesn't support it.

Too much coding.  Yes, platform().
Ok, what are the values though?  If you don't want to list them, then maybe just link to this:  http://cordova.apache.org/docs/en/2.4.0/cordova_device_device.md.html#device.platform
Also, which ones does ionic support then?
Thanks!

Well again, it depends on the device plugin, which supports all of cordovas platforms.
https://github.com/apache/cordova-plugin-device#devicecordova
.platform() is essentially a alias for cordova's device.platform.
So ionic only supports iOS and Android for the whole framework, but ionic.Platform.platform(); would technically work on all corodva's platforms.


Add ability to read CSV data files

In the eva/data sub-directory add a module to enable reading in CSV formatted text files named csv_space.py.  This is needed in order to generate the MinMon summary plots.
The idea is to read the data in a generic fashion so any CSV file could be read into eva.  The input yaml file will indicate which field(s) in the file are to be to incorporated in the resulting dataset, as well as the names of the corresponding variables.

Just about there.  This plot is taken from the gnorm_data.txt file, which is in CSV format.  This 3 figure plot is MinMon summary plot and includes the initial gradient over the past 120 cycles, the final gnorm value, and the minimum and maximum values over the last 10 iterations.


Automatically GC thread storage

Check the return from pthread_create correctly, and detach threads to let the system reclaim storage.
Previously to this, cbridge would hang after less than 500 outgoing NCP connections (on my Raspi system).

@ericswenson1 please try it out and see whether your problems also go away?

I have updated my 2 cbridge instances to the latest.  But I hadn't had any issues in the last week (since I wrote you about them), so I don't know that I would notice the fix.  But I'll let you know if I have any issues with this latest version. Thanks.


Android some errors

Hello @rossmartin ,
On Android, when I run onCancelNavigation or close the page. It gives errors while unmounting.
I think the problem will be resolved if the OnDestroy code is integrated. Can you fix this?

Please see here https://github.com/homeeondemand/react-native-mapbox-navigation/issues/2#issuecomment-744041044 and https://github.com/homeeondemand/react-native-mapbox-navigation/issues/2#issuecomment-763193599
It will only happen in a release build. I'll add that to the README in a future release.

@rossmartin Oh, okay man thank you.


Project ideas

Thread for suggesting topics for our dashboard and linking relevant data sets we could use.

Potential dataset to work with: Dogs: https://github.com/rfordatascience/tidytuesday/tree/master/data/2022/2022-02-01

Per meeting today, we have the consensus to use this dataset for creating the dashboard.


Investigate failing tests on bots

The following tests are failing in dartium on the bots, but work locally.

build/test/publish_attributes_test
build/test/publish_inherited_properties_test

looks like this is caused by https://github.com/dart-lang/package-bots/issues/2


How can I set DDL_LOCK_TIMEOUT before resetPassword？

Hi @gvenzl
The Apache Spark Docker Integration tests sometimes fail due to the failure of docker to start. It seems like the reason behind this is ORA-04021. To fix the problem, we can try to increase DDL_LOCK_TIMEOUT. Can we set this value using environment variables?
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: CONTAINER: Resetting SYS and SYSTEM passwords.
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT:
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: User altered.
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT:
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ALTER USER SYSTEM IDENTIFIED BY "***************"
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: *
24/01/04 04:22:09.284 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ERROR at line 1:
24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ORA-65048: error encountered when processing the current DDL statement in
24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: pluggable database FREEPDB1
24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: ORA-04021: timeout occurred while waiting to lock object
24/01/04 04:22:09.285 docker-java-stream-568158377 INFO OracleIntegrationSuite: STDOUT: Help: https://docs.oracle.com/error-help/db/ora-65048/

Hey @yaooqinn,
Thanks a lot for using these images for the Apache Spark integration tests!
Yeah, I have no problem increasing the timeout (already saw the PR, thanks!) but I'm curious as to why this error is happening.
Technically, not much should be happening on the database yet that would cause a DDL lock timeout.
Are the Spark jobs connecting via the SYS user at this stage already and creating some additional users or something like that?

Thank you @gvenzl.
Spark's docker IT uses docker-java library to start an oracle-free:23.3 container. After setting up some specs, it first uses startContainerCmd to launch the container and waits in a loop with inspectContainerCmd for the running state. If it detects the container is running, then it uses a time-limited loop with an interval of 1 second to wait for the JDBC connection to be established.

Technically, not much should be happening on the database yet that would cause a DDL lock timeout.

Would establishing the connection within an interval of 1 second be considered pushy and cause the DDL LOCK problem?

Are the Spark jobs connecting via the SYS user at this stage already and creating some additional users or something like that?

The SYSTEM user identifier is used for connections, not SYS. No additional users are being created.

Hey @yaooqinn,
Thanks a lot for the background, this is interesting.

Technically, not much should be happening on the database yet that would cause a DDL lock timeout.

Would establishing the connection within an interval of 1 second be considered pushy and cause the DDL LOCK problem?

No, an interval of 1 second shouldn't be a problem at all, and actually, it cannot cause this error. ORA-04021: timeout occurred while waiting to lock object means that something has a lock on the table while something else is trying to change it. If there is no connection to the database yet, the connection cannot hold a lock.

Are the Spark jobs connecting via the SYS user at this stage already and creating some additional users or something like that?

The SYSTEM user identifier is used for connections, not SYS. No additional users are being created.

Hm, even more peculiar. Given that the SYSTEM password reset fails with this error, I can't see how anything in the test would cause that error, as no test would be able to logon due to the not-yet-reset password.
The more I think about this, the more I think we have found some sort of race condition or bug here. Is there any way for me to see what the regression tests execute, what's done inside inspectContainerCmd and perhaps for me to clone the repo and try to reproduce this myself?

Thank you @gvenzl.
Here is I recent example I have found:
https://github.com/apache/spark/actions/runs/7749877156/job/21135223061
To reproduce this in GitHub Actions, you can fork the spark repo, and the Run / Run Docker integration tests job will run in your forked repo by push events.
To reproduce locally,
ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:23.3 ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite"

Here is a detailed guide you may refer to: https://github.com/apache/spark/tree/master/connector/docker-integration-tests#readme
FYI, the code for docker container initialization can be found at:
https://github.com/apache/spark/blob/e3fd64b96765dc04994798e79161026bd1361230/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/DockerJDBCIntegrationSuite.scala#L182

When the GitHub action job fails, you can found the error log on the Summary page , such as, unit-tests-log-docker-integration--17-hadoop3-hive2.3

Thanks a lot, @yaooqinn!
Let me give this a spin and take a look what's going on here.

Thank you very much for the help @gvenzl

Ok, I got it running locally.
To no big surprise, so far I haven't managed to reproduce it yet.
However, I have looked through a bunch of failed tests and seem to find a lot more runs that simply time out with:

[info] OracleIntegrationSuite:
[info] org.apache.spark.sql.jdbc.OracleIntegrationSuite *** ABORTED *** (7 minutes, 31 seconds)
[info]   The code passed to eventually never returned normally. Attempted 418 times over 7.002120891733333 minutes. Last failure message: ORA-12541: Cannot connect. No listener at host <IP_ADDRESS> port 43305. (CONNECTION_ID=7sVBltRxQr+1iXsn3hBBtQ==)

I'll keep digging.

Besides the console log, much richer information, including those from docker container, could be found at connector/docker-integration-tests/target/unit-tests.log

Thanks a lot, @yaooqinn,
Btw, I don't think that the workaround is an actual workaround.
According to
https://github.com/gvenzl/spark/blob/07f1341fac6ff23f1c5a30c87a0d9946ca794e61/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala#L47-L59
The DDL_LOCK_TIMEOUT is set as an init script which is executed after the initial ALTER USER, i.e. the workaround sets the DDL_LOCK_TIMEOUT after the two User altered. statements:
ORACLE instance started.

Total System Global Area<PHONE_NUMBER> bytes
Fixed Size		    5313720 bytes
Variable Size		  738197504 bytes
Database Buffers	  855638016 bytes
Redo Buffers		    4530176 bytes
Database mounted.
Database opened.

CONTAINER: Resetting SYS and SYSTEM passwords.

User altered.

User altered.

CONTAINER: Executing user-defined scripts...

CONTAINER: running /docker-entrypoint-initdb.d/install.sql ...

Session altered.

User altered.

CONTAINER: DONE: running /docker-entrypoint-initdb.d/install.sql

CONTAINER: DONE: Executing user-defined scripts.

#########################
DATABASE IS READY TO USE!
#########################

Yet, the original error occurred during the second User altered. and hence before the init scripts are executed:
CONTAINER: Resetting SYS and SYSTEM passwords.

User altered.

ALTER USER SYSTEM IDENTIFIED BY "***************"
*
ERROR at line 1:
ORA-65048: error encountered when processing the current DDL statement in
pluggable database FREEPDB1
ORA-04021: timeout occurred while waiting to lock object
Help: https://docs.oracle.com/error-help/db/ora-65048/

Hey @yaooqinn,
Is there any way to see or extract the container logs in cases such as this one:
[info] org.apache.spark.sql.jdbc.OracleIntegrationSuite *** ABORTED *** (7 minutes, 31 seconds)
[info]   The code passed to eventually never returned normally. Attempted 418 times over 7.002120891733333 minutes. Last failure message: ORA-12541: Cannot connect. No listener at host <IP_ADDRESS> port 43305. (CONNECTION_ID=7sVBltRxQr+1iXsn3hBBtQ==)
[info]   [https://docs.oracle.com/error-help/db/ora-12541/.](https://docs.oracle.com/error-help/db/ora-12541/) (DockerJDBCIntegrationSuite.scala:189)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at org.scalatest.enablers.Retrying$$anon$4.tryTryAgain$2(Retrying.scala:219)
[info]   at org.scalatest.enablers.Retrying$$anon$4.retry(Retrying.scala:226)
[info]   at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:313)
[info]   at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:312)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.eventually(DockerJDBCIntegrationSuite.scala:99)
[info]   at org.apache.spark.sql.jdbc.DockerJDBCIntegrationSuite.$anonfun$beforeAll$1(DockerJDBCIntegrationSuite.scala:189)
[info]   at org.apache.spark.sql.jdbc.DockerIntegrationFunSuite.runIfTestsEnabled(DockerIntegrationFunSuite.scala:49)

Thanks @yaooqinn,
Unfortunately, neither of these are very helpful.
My latest suspicion is that the runner simply runs out of either horsepower, i.e. CPU, or memory.
When I ran these tests locally on a VM with only 4G of memory, it wasn't pretty :)
So I was wondering what would make the database not be able to spin up a listener or take forever to run a simple ALTER USER command.
Nothing really should, as no activity on the DB has been executed to that stage, it's just starting up. So I was wondering whether the runner is simply overloaded.
From what I've found out, these runners come with 4 CPUs only, and 16 GB memory, which ought to be enough to run these tests, unless there are some executed in parallel.
I've added a couple of checks for the memory and CPU before the stage where no tests have been run yet. Obviously, that's not very helpful but at least it shows what the runner has available: https://github.com/gvenzl/spark/actions/runs/8015324674/job/21895352666 (see Run --> Run Docker integration tests --> Free memory & CPU stats staged)
It seems like running these tests eats up about 3 - 3.5GB of RAM:
Here is the output from a run with ORACLE_DOCKER_IMAGE_NAME=gvenzl/oracle-free:23.3 ./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly *OracleIntegrationSuite":
[gvenzl@localhost ~]$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 13214936   1628 2454320    0    0  1986  4577 1027  798 13 14 70  3  0
 3  0      0 12933988   1628 2467436    0    0  4359     0 2555 1986 12  8 79  1  0
 3  0      0 12704384   1628 2468092    0    0   221     0 5424 2850 49 20 30  1  0
 4  0      0 12404572   1628 2481192    0    0  4367   448 6482 3243 41 29 30  0  0
 2  0      0 12373512   1628 2481812    0    0   148     0 4817 2068 50 17 32  0  0
 2  0      0 12284284   1628 2481812    0    0     0    12 4194 1889 41 18 41  0  0
 3  0      0 11970348   1628 2481888    0    0    13  2160 6102 5542 46 29 25  0  0 <-- compilation starts up
 4  0      0 11901908   1628 2481928    0    0    19     0 5073 2778 47 19 33  0  0
 1  0      0 11700828   1628 2481928    0    0     0   887 4241 2020 36 17 47  0  0
 2  0      0 11265520   1628 2482028    0    0    35   936 4075 1928 35 14 51  0  0
 2  0      0 10875900   1628 2482032    0    0     0  1028 3649 1875 27 13 61  0  0
 6  0      0 10531076   1628 2483464    0    0<PHONE_NUMBER> 7186 34 32 34  0  0
11  0      0 10333472   1628 2503632    0    0  6768   859 7948 6059 36 58  6  0  0
 6  0      0 10166876   1628 2542188    0    0 12805   181 7390 1878 22 46 31  0  0
 7  0      0 9856304   1628 2569000    0    0  9047    55 7244 3249 36 45 19  0  0
 1  0      0 9351240   1628 2583280    0    0  4615   146 5543 2739 27 22 51  1  0
 4  0      0 8777552   1628 2583280    0    0     0    53 2445  963 16  9 75  0  0
 1  0      0 8716568   1628 2583312    0    0     0     0 3453 1061 31 14 55  0  0
 2  0      0 8590052   1628 2583328    0    0     0    39 1838  788  8  7 85  0  0
 8  0      0 8224272   1628 2583620    0    0     0    60 6060 4179 45 30 25  0  0
 4  0      0 8077572   1628 2583620    0    0     0     3 4931 2805 47 32 21  0  0
 3  0      0 8002792   1628 2583624    0    0     0    75 4081 2088 32 16 51  0  0  <-- First Oracle DB docker container is started (note that a lot of memory is allocated to the Linux FS cache, not necessarily not taken away)
 5  0      0 6827184   1628 3387180    0    0  5581   768 5048 2785 33 40 26  1  0
 5  1      0 5646408   1628 4550088    0    0  1319<PHONE_NUMBER>  998 34 38 17 10  0
 4  1      0 5066112   1628 5098800    0    0     0<PHONE_NUMBER>  807 38 46  2 13  0
 0  1      0 4844880   1628 5611496    0    0   821<PHONE_NUMBER> 2902 14 23 49 13  0
 0  0      0 4842696   1628 5611668    0    0  1931  3427 5922 7297  1 15 77  7  0
 1  0      0 4630312   1628 5820208    0    0   629<PHONE_NUMBER> 4286  2 14 68 16  0
 4  0      0 2965884   1628 7167516    0    0   561  1186 3750 2324 10 17 69  3  0
 1  0      0 2844004   1628 7167928    0    0     4  2294 5250 4316 34 23 38  6  0
 2  0      0 2797840   1628 7168176    0    0     0  2312 4987 4693 26 18 49  6  0
 3  0      0 2744224   1628 7168044    0    0     0   109 4059 2484 19 17 63  0  0
 9  0      0 2692996   1628 7168052    0    0     0   924 4673 3094 32 20 47  1  0
 5  0      0 2681664   1628 7168388    0    0     0   820 4772 3102 23 21 54  2  0
 3  0      0 2631880   1628 7168444    0    0    11    72 4755 3393 31 16 53  0  0
 3  0      0 2555164   1628 7168560    0    0     0   341 5438 4133 37 20 43  1  0
 3  0      0 2416756   1628 7168728    0    0    13 10673 5054 3783 37 21 42  0  0
 4  0      0 2289964   1628 7168852    0    0     0    88 5854 5425 25 20 54  0  0
 3  0      0 7295732   1628 2606544    0    0   165  1109 5357 5365 12 18 69  1  0 <-- second Oracle DB container starts up (again, deallocation and allocation from Linux FS cache)
 4  2      0 5536388   1628 4133824    0    0     8<PHONE_NUMBER> 1247 31 24 38  7  0
 5  0      0 4620440   1628 5035524    0    0     0<PHONE_NUMBER>  584 39 28  9 25  0
 0  1      0 4364196   1628 5613288    0    0     8<PHONE_NUMBER>  896 17 15 45 23  0
 0  0      0 4350664   1628 5613524    0    0  2255<PHONE_NUMBER> 5435  1 14 69 17  0
 0  0      0 4132684   1628 5822236    0    0   757  3105 5355 6371  1 13 79  6  0
 1  0      0 2514348   1628 7170088    0    0   991 33561 4869 4254  8 18 65  9  0
 3  0      0 2359032   1628 7170556    0    0     8  2082 4655 3460 26 20 50  4  0
 7  1      0 2280500   1628 7170844    0    0     0  2683 5875 5548 27 21 46  6  0
 9  0      0 2164488   1628 7171032    0    0     0   628 5245 3588 40 27 32  1  0
 4  0      0 2135056   1628 7171304    0    0     0   593 5164 3743 36 24 39  1  0
 0  0      0 13053436   1628 2607480    0    0     0 12603 4258 5729 10 15 74  1  0
 0  0      0 13052744   1628 2607448    0    0     0     0  125  150  0  1 99  0  0

With these tests taking about 3 - 3.5 GB of memory and Oracle DB taking about another 1.5 ~ 2 GB, an active run would worst case take about 5.5GB, which in itself shouldn't be an issue, unless other stuff runs in parallel on the machine.
Same with the CPU, the run queue is somewhere around nothing to 4/5 ish, again, not a biggy, unless other things run in parallel, perhaps.
What would be really interesting is if we could inject these resource verification steps while these tests are running.
Would you happen to know an easy way to do that?
Or do you know whether anything else is running in parallel on the same runner?

I forgot to add that so far I was unable to reproduce this locally on my VM , another reason why I'm suspicious whether this is an issue of too many things running in parallel during the GHA executions.

Thank you for the detailed information.

Would you happen to know an easy way to do that?

Unfortunately, I don't know.

Or do you know whether anything else is running in parallel on the same runner?

There are no parallel test works being executed.
Given on my observations, the error in GHA always/coincidentally occured on the second OracleIntegrationSuite.

I forgot to add that so far I was unable to reproduce this locally on my VM , another reason why I'm suspicious whether this is an issue of too many things running in parallel during the GHA executions.

The issue can be reproduced my local Apple M2/colima docker context, it should not be GHA-specific.

If the load is the root cause, changing the ddl lock timeout seems to be a positive change https://github.com/gvenzl/oci-oracle-free/pull/37？

Would you happen to know an easy way to do that?

I have discovered that we can include a health check to ensure that we achieve this goal. https://github.com/docker-java/docker-java/pull/2244/files

Hey @yaooqinn,
Thanks, I wasn't aware that you can replicate it in your local environment.
Then let me take back my theory regarding the resource consumption. I think you would notice if your laptop would come to a grinding halt and it should have more than enough horsepower to run tests even in parallel.
The DDL_LOCK_TIMEOUT won't achieve anything, unfortunately.
The error Last failure message: ORA-12541: Cannot connect. No listener at host <IP_ADDRESS> port 43305. (CONNECTION_ID=7sVBltRxQr+1iXsn3hBBtQ==) means that the program couldn't even connect to the database.
The question remains as to why.
When you encounter this error locally, could you please provide me with the log from the currently running Oracle Database container via (docker logs <container name>)?
I have still not managed to reproduce this issue at my end, but will continue to try.

I have discovered that we can include a health check to ensure that we achieve this goal. https://github.com/docker-java/docker-java/pull/2244/files

Hm, I'm not sure what exactly that addition does but it sounds like it only provides for a start interval delay for running the healthcheck. What we need really is the container's log from when that failure ocurrs, do you think this will provide us with that?
Thanks,

The DDL_LOCK_TIMEOUT won't achieve anything, unfortunately.
The error Last failure message: ORA-12541: Cannot connect. No listener at host <IP_ADDRESS> port 43305. (CONNECTION_ID=7sVBltRxQr+1iXsn3hBBtQ==) means that the program couldn't even connect to the database.
The question remains as to why.

It seems that ORA-12541 can be ignored, the normal ones also include such errors

Ok, although I saw test failures on the official Spark repository actions with this error.
In any case, if you could provide me with the container log, it would already help tremendously.
Even more so if you could also copy the following file out from the container (the database main log): /opt/oracle/diag/rdbms/free/FREE/trace/alert_FREE.log

24/03/05 14:05:37.470 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: starting up...
24/03/05 14:05:37.470 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: first database startup, initializing...
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: uncompressing database data files, please wait...
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: done uncompressing database data files, duration: 52 seconds.
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: starting up Oracle Database...
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: LSNRCTL for Linux: Version <IP_ADDRESS>.0 - Production on 05-MAR-2024 06:02:01
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Copyright (c) 1991, 2023, Oracle.  All rights reserved.
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Starting /opt/oracle/product/23c/dbhomeFree/bin/tnslsnr: please wait...
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: TNSLSNR for Linux: Version <IP_ADDRESS>.0 - Production
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: System parameter file is /opt/oracle/product/23c/dbhomeFree/network/admin/listener.ora
24/03/05 14:05:37.483 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Log messages written to /opt/oracle/diag/tnslsnr/4a50261d8539/listener/alert/log.xml
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<IP_ADDRESS>)(PORT=1521)))
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_FREE)))
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: STATUS of the LISTENER
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ------------------------
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Alias                     LISTENER
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Version                   TNSLSNR for Linux: Version <IP_ADDRESS>.0 - Production
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Start Date                05-MAR-2024 06:02:01
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Uptime                    0 days 0 hr. 0 min. 0 sec
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Trace Level               off
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Security                  ON: Local OS Authentication
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: SNMP                      OFF
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Default Service           FREE
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Listener Parameter File   /opt/oracle/product/23c/dbhomeFree/network/admin/listener.ora
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Listener Log File         /opt/oracle/diag/tnslsnr/4a50261d8539/listener/alert/log.xml
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Listening Endpoints Summary...
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<IP_ADDRESS>)(PORT=1521)))
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: The listener supports no services
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: The command completed successfully
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ORACLE instance started.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Total System Global Area<PHONE_NUMBER> bytes
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Fixed Size		    5313720 bytes
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Variable Size		  738197504 bytes
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Database Buffers	  855638016 bytes
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Redo Buffers		    4530176 bytes
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Database mounted.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Database opened.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: Resetting SYS and SYSTEM passwords.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: User altered.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: User altered.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: Executing user-defined scripts...
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: running /docker-entrypoint-initdb.d/install.sql ...
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Session altered.
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ALTER USER SYSTEM IDENTIFIED BY "Th1s1sThe0racle#Pass"
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: *
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ERROR at line 1:
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ORA-65048: error encountered when processing the current DDL statement in
24/03/05 14:05:37.484 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: pluggable database FREEPDB1
24/03/05 14:05:37.500 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ORA-04021: timeout occurred while waiting to lock object
24/03/05 14:05:37.502 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Help: https://docs.oracle.com/error-help/db/ora-65048/
24/03/05 14:05:37.503 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.504 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.505 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: DONE: running /docker-entrypoint-initdb.d/install.sql
24/03/05 14:05:37.506 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.507 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: DONE: Executing user-defined scripts.
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: #########################
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: DATABASE IS READY TO USE!
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: #########################
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT:
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ####################################################################
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: The following output is now from the alert_FREE.log file:
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ####################################################################
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Dumping current patch information
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ===========================================================
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Unable to obtain current patch information due to error: 20013
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: ===========================================================
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: FREEPDB1(3):Autotune of undo retention is turned on.
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: 2024-03-05T06:03:11.692486+00:00
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: FREEPDB1(3):Opening pdb with Resource Manager plan: DEFAULT_PLAN
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: 2024-03-05T06:03:15.503934+00:00
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Completed: Pluggable database FREEPDB1 opened read write
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Completed: ALTER DATABASE OPEN
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: 2024-03-05T06:04:09.299018+00:00
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Thread 1 advanced to log sequence 14 (LGWR switch),  current SCN: 2337234
24/03/05 14:05:37.509 docker-java-stream-500438131 INFO OracleIntegrationSuite: STDOUT: Current log# 1 seq# 14 mem# 0: /opt/oracle/oradata/FREE/redo01.log

The Java API was used to collect the logs above, which are expected to be the same as the output of the docker logs command.

Thanks a lot, @yaooqinn.
Is there any reason why you are resetting the SYSTEM password twice?
The SYSTEM password has already been reset (successfully) just about the user-defined scripts:
CONTAINER: Resetting SYS and SYSTEM passwords.

User altered.

User altered.

CONTAINER: Executing user-defined scripts...
...

Sorry, I just found it here:
https://github.com/gvenzl/spark/blob/5fa42804326f41e5d107a787d96fd8d7bcf4a215/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala#L47-L56
That also confirms again that the workaround of increasing the DDL_LOCK_TIMEOUT to a higher number won't remedy the situation either, as the ALTER SESSION SET DDL_LOCK_TIMEOUT=30 has just been processed before the second ALTER USER SYSTEM IDENTIFIED ...

Hi @gvenzl
I have witnessed failures even with DDL_LOCK_TIMEOUT=30, indicating its ineffectiveness.
I have no idea how to fix this:(

Hey @yaooqinn,
I chatted about this with my colleagues a bit and there is no easy way to diagnose this without getting trace files from the database to know where this is happening.
This would require some magic as part of the init script, basically, it would have to be a script that enables tracing and, when this error occurs, looks for the trace file and writes it to STDOUT so that we can see it via the container logs.
I can help craft such a init script, but a fair warning, it will probably require a couple of round trips.
However, before we go there, Oracle just released a new version of Oracle Database Free (23.4) last Thursday, and I updated my images last night.
So, in the slight chance that this was some internal issue that was fixed in 23.4 in the meantime, could we first try and have Apache Spark run against the latest gvenzl/oracle-free:23.4-slim image?
I don't know how much effort that is, ideally just a simple switch of one digit in https://github.com/apache/spark/blob/bf2e25459fe46ca2b1d26e1c98c873923fc135e1/connector/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleDatabaseOnDocker.scala#L29 and nothing else.

Thank you @gvenzl
I have seen a PR https://github.com/apache/spark/pull/46399 for bumping up to 23.4

I recommend removing the code that sets the password in override def beforeContainerStart. As @gvenzl pointed out, this step is redundant because setting the environment variable ORACLE_PASSWORD already configures the SYSTEM password as well. I've conducted a preliminary test and confirmed that the functionality remains intact without this segment of code. If there are no objections, I'm ready to submit a pull request to Apache Spark to implement this change.

I have created a PR for this , see https://issues.apache.org/jira/browse/SPARK-48289 basically reverting the extra password change and DDL_LOCK_TIMEOUT. It does look like we hit a bug / unwanted feature ;) in Oracle (maybe related to PDB?) that cannot be easily worked around with the attempted change (good try BTW). So my view is that the extra code can be removed as it creates extra space for issues. Our hope at this stage is that the issue is not there in Oracle Free 23.4. Otherwise we can keep investigating.

Hey @LucaCanali, thanks for your help on this!
If this issue reoccurs, it would be great to have a reproducible test case, ideally outside of the Spark test runs via a "pure" Docker + scripts environment as that would make it easier for me to debug.
But hopefully, this is solved now.

Thanks @gvenzl and @LucaCanali

But hopefully, this is solved now.

I haven't seen that error in a long time since the update. Everything seems fine now.

Oops, the error is still there to make the test flakey https://github.com/yaooqinn/spark/actions/runs/9155473477/job/25167935142

I can see the error:
[info] org.apache.spark.sql.jdbc.OracleIntegrationSuite *** ABORTED *** (5 minutes, 17 seconds)
[info]   The code passed to eventually never returned normally. Attempted 299 times over 5.00881802855 minutes. Last failure message: ORA-12541: Cannot connect. No listener at host <IP_ADDRESS> port 46019. (CONNECTION_ID=IN3RqPhUS3Kha0Wdff/dPg==)

This issue might be related to port forwarding. Based on the error message, it appears that IP:PORT <IP_ADDRESS>:46019 should be forwarded to the container's port 1521, where the listener runs. If this forwarding is not functioning correctly, it could explain the ORA-12541 error. Another possibility is that the listener did not start up properly within the container, though this seems less likely to me.

@LucaCanali You can find more information about the issue by referring to the above discussions

@yaooqinn, I reviewed the logs from the following job run:   https://github.com/yaooqinn/spark/actions/runs/9155473477/job/25167935142 is The error identified in the logs is ORA-12541: TNS:no listener, indicating that the client cannot connect to the Oracle service on host <IP_ADDRESS> at port 46019.
This error seems to be unrelated to the previously reported errors ORA-65048 and ORA-04021. ORA-04021, as previously discussed in this thread, is usually associated with a timeout waiting for a resource, such as a library cache lock (when changing the password for example). ORA-65048 typically relates to an issue with attempting to access an invalid or non-existent container within a pluggable database environment. These errors suggest different root causes compared to the current connectivity issue.
Additionally, could we gain insights into the current frequency of failures with the OracleIntegrationSuite? Are there statistics or logs available that detail these occurrences? Moreover, is there a way to assess if the testing infrastructure might be experiencing high CPU load, potentially leading to slow container image start times? This information could be crucial for identifying recurring patterns or persistent issues.
Thank you!

Hi @LucaCanali. I am sorry that I attached the wrong link as I was filing multiple PRs to spark repo at that time, it shall be this one - https://github.com/yaooqinn/spark/actions/runs/9155585688/job/25169246357
It's still ORA-65048 with ORA-04021 as the root cause
===== END OF CONTAINER LOGS FOR container Id: CreateContainerResponse(id=3c6690ffa72f630c79bdde515d9a6a9128dacc3fabae20df3713b670d0065d8d, warnings=[]) =====
24/05/20 09:20:40.045 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: starting up...
24/05/20 09:20:40.045 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: first database startup, initializing...
24/05/20 09:20:40.045 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: uncompressing database data files, please wait...
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: done uncompressing database data files, duration: 7 seconds.
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: starting up Oracle Database...
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: LSNRCTL for Linux: Version <IP_ADDRESS>.0 - Production on 20-MAY-2024 09:15:46
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Copyright (c) 1991, 2024, Oracle.  All rights reserved.
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Starting /opt/oracle/product/23ai/dbhomeFree/bin/tnslsnr: please wait...
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: TNSLSNR for Linux: Version <IP_ADDRESS>.0 - Production
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: System parameter file is /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Log messages written to /opt/oracle/diag/tnslsnr/3c6690ffa72f/listener/alert/log.xml
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<IP_ADDRESS>)(PORT=1521)))
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC_FOR_FREE)))
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: STATUS of the LISTENER
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: ------------------------
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Alias                     LISTENER
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Version                   TNSLSNR for Linux: Version <IP_ADDRESS>.0 - Production
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Start Date                20-MAY-2024 09:15:46
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Uptime                    0 days 0 hr. 0 min. 0 sec
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Trace Level               off
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Security                  ON: Local OS Authentication
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: SNMP                      OFF
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Default Service           FREE
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Listener Parameter File   /opt/oracle/product/23ai/dbhomeFree/network/admin/listener.ora
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Listener Log File         /opt/oracle/diag/tnslsnr/3c6690ffa72f/listener/alert/log.xml
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Listening Endpoints Summary...
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC_FOR_FREE)))
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<IP_ADDRESS>)(PORT=1521)))
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: The listener supports no services
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: The command completed successfully
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: ORACLE instance started.
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Total System Global Area<PHONE_NUMBER> bytes
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Fixed Size		    5360648 bytes
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Variable Size		  788529152 bytes
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Database Buffers	  805306368 bytes
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Redo Buffers		    4530176 bytes
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Database mounted.
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Database opened.
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.046 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: CONTAINER: Resetting SYS and SYSTEM passwords.
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: User altered.
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: ALTER USER SYSTEM IDENTIFIED BY "Th1s1sThe0racle#Pass"
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: *
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: ERROR at line 1:
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: ORA-65048: error encountered when processing the current DDL statement in
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: pluggable database FREEPDB1
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: ORA-04021: timeout occurred while waiting to lock object
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT: Help: https://docs.oracle.com/error-help/db/ora-65048/
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:
24/05/20 09:20:40.047 docker-java-stream-1531082131 INFO OracleIntegrationSuite: STDOUT:

Additionally, could we gain insights into the current frequency of failures with the OracleIntegrationSuite?

As Spark runs specific CI jobs by identifying related code change, it depends on how often we change the JDBC connectors. I have been working on an umbrella https://issues.apache.org/jira/browse/SPARK-47361 that touches this part frequently, so based on my observation, the rate is very high

Are there statistics or logs available that detail these occurrences?

Unfortunately, No.

Moreover, is there a way to assess if the testing infrastructure might be experiencing high CPU load, potentially leading to slow container image start times?

ORA-04021 can be reproduced on local machines, not necessary has to be testing infrastructures like Github Actions. I'm not sure whether ORA-04021's related to high CPU load or not.

Oh, I see. You only viewed the console log but not the unit-tests.log. The console log's level is OFF, so you can not see the underlying root cause. I noticed ORA-12541, but I ignored it because I thought you were referring to the 'unit-tests.log' as well. ORA-12541 is a client-side error, indicating that the database instance or container has been terminated due to ORA-04021 in this case.

Thank you for the explanation. I can see that you had ORA-0421 now in https://github.com/yaooqinn/spark/actions/runs/9155585688/job/25169246357
I have been trying to reproduce the error locally by running the tests a few times, but it did not appear in my system
export ENABLE_DOCKER_INTEGRATION_TESTS=1
./build/sbt -Pdocker-integration-tests "docker-integration-tests/testOnly org.apache.spark.sql.jdbc.OracleIntegrationSuite"

I am now trying to reproduce the issue using Github actions.

I have been attempting to reproduce the issue using GitHub Actions. However, the OracleIntegrationSuite has been running successfully there as well, so far.

Hi @yaooqinn, @LucaCanali,
@yaooqinn, is this still an issue at your end?

Yes. @gvenzl


OHIE-155 doc manifest

Current coverage is 60.16% (diff: 79.59%)

Merging #5 into master will increase coverage by 1.30%

@@             master         #5   diff @@
==========================================
  Files            21         22     +1
  Lines          1055       1092    +37
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits            621        657    +36
- Misses          434        435     +1
  Partials          0          0

Powered by Codecov. Last update 3d23a64...82f7739

Closing this  PR as these changes were merged with the changes in https://github.com/jembi/hearth/pull/4 as there was overlap with parts of the code. Please see that PR for the full, combined set of changes.


ImportError: This version of TensorFlow Datasets requires TensorFlow version >= 2.1.0; Detected an installation of version 1.15.0. Please upgrade TensorFlow to proceed.

Traceback (most recent call last):
File "run_low_shot.py", line 16, in
from metrics import metric_base
File "/content/data-efficient-gans/DiffAugment-stylegan2/metrics/metric_base.py", line 18, in
from training import dataset
File "/content/data-efficient-gans/DiffAugment-stylegan2/training/dataset.py", line 14, in
import tensorflow_datasets as tfds
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/init.py", line 51, in
from tensorflow_datasets import __init__py3 as api
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/__init__py3.py", line 43, in
from tensorflow_datasets.core import tf_compat
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/init.py", line 21, in
tf_compat.ensure_tf_install()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_datasets/core/tf_compat.py", line 59, in ensure_tf_install
"This version of TensorFlow Datasets requires TensorFlow "
ImportError: This version of TensorFlow Datasets requires TensorFlow version >= 2.1.0; Detected an installation of version 1.15.0. Please upgrade TensorFlow to proceed.

Please try pip install tensorflow-datasets==2.1.0

Please try pip install tensorflow-datasets==2.1.0

ok thanks!


exclude hidden git folder when detecting nwb files

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 1 (0.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 72.316%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

bep032tools/generator/nwb2bidsgenerator.py
0
1
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
896

Relevant Lines:
1239

💛  - Coveralls


Need help.

I don't know if it's a bug or what but if I use next_message it returns none as response. Any help?

the issue is outdated, check on the new version


RuntimeEncoder and RuntimeDecoder produce invalid SymbolicPulses with terra 0.22

Describe the bug
Using the RuntimeEncoder and RuntimeDecoder on a circuit with a gate calibration produces invalid pulses when the calibration includes one of the SymbolicPulse shapes included in qiskit-terra.
Steps to reproduce
import json

from qiskit import QuantumCircuit, assemble, pulse
from qiskit_ibm_runtime.utils import RuntimeEncoder, RuntimeDecoder

with pulse.builder.build(name="x") as sched:
    pulse.play(pulse.Gaussian(100, 0.8, 10), pulse.DriveChannel(0))

qc = QuantumCircuit(1)
qc.x(0)
qc.add_calibration("x", (0,), sched)

dumped = json.dumps(qc, cls=RuntimeEncoder)
loaded = json.loads(dumped, cls=RuntimeDecoder)

print(type(qc.calibrations["x"][((0,),())].instructions[0][-1].pulse))
print(type(loaded.calibrations["x"][((0,),())].instructions[0][-1].pulse))

This prints:
<class 'qiskit.pulse.library.symbolic_pulses.SymbolicPulse'>
<class 'qiskit.pulse.library.symbolic_pulses.Gaussian'>

Expected behavior
Both versions of the pulse should be SymbolicPulse.
Suggested solutions
I believe that #577 would fix the problem though I did not test it. I did test that doing a direct dump and load with qpy from qiskit-terra produced the correct class for the pulse.
The issue is that qiskit-terra 0.22 deprecated the SymbolicPulse subclasses by changing them to define a __new__ constructor that generates a SymbolicPulse object directly. The 0.21 qpy loader creates the SymbolicPulse subtype directly with object.__new__() and sets its attributes without invoking the class's constructor. With the changes in 0.22, this loader now generates a metaclass that does not function as a proper SymbolicPulse class.
Additional notes
@kevinsung ran into this issue trying to run a job with pulse gates using qiskit-terra 0.22. What I think happens is that when the backend code loads the qpy payload and assembles it into a qobj, it hits this conditional that checks for a SymbolicPulse and skips over it, skipping pulling the pulse parameters out of the object. The backend ends up returning a "Pulse name is not a string" error because the pulse name is left as None.
One thing I think might merit further discussion is how to avoid this kind of issue. I believe that #577 is backwards compatible -- I think it will work even with qiskit-terra 0.21 (and maybe earlier). However, the changes made in qiskit-terra 0.22 were not compatible with the 0.21 qpy loader in the runtime package. Is there a way to update qpy in terra without breaking the runtime package using the previous version? Would it help if qpy were its own package and terra committed to being compatible with more than one version at a time? @jakelishman @mtreinish @nkanazawa1989
Final note -- there is also #655 which is a similar symptom of the same issue, I think, but runs into a different error.
Additional Information

qiskit-ibm-runtime version: 0.8.0
qiskit-terra version: 0.22.3
Python version: 3.10.6
Operating system: Fedora 36

Possible workaround suggested by Will:

The best you might be able to do is convert all the pulses into Waveforms. Another option is to go through and overwrite _pulse_type on the pulses so that it is not Drag or GaussianSquare. Then the pulses will get serialized as parametric pulses (instead of waveform) but in the runtime when they are deserialized they will not be recongized pulse types and will be converted to Waveforms.

Terra already commits to complete backwards compatibility with QPY, so new Terra will always told QPY generated by old Terra. What you're suggesting here is forwards compatibility, i.e. that we should constrain Terra's development so much that old Terra can load new objects. That's not reasonably possible for a library that needs to evolve like Terra.
The solution to me is just for Runtime to update their vendored version of QPY a little faster. Given it's vendored for stability, and Terra's backwards compatible guarantees, Runtime could proactively update QPY from Terra main, but at the moment it's still using QPY loading code from Terra 0.21, while telling people to use 0.22 to generate the circuits.

What you're suggesting here is forwards compatibility, i.e. that we should constrain Terra's development so much that old Terra can load new objects that might not have existed when the loader code was written (like this case).

No, that is not quite what I was suggesting. In particular, you reference "old Terra" but the issue we are talking about here involves new terra on both the client and the server (I used terra 0.22 and runtim 0.8.0 and I believe the server is using those as well). "qpy" is both a binary format for storing quantum circuits and a Python library for storing and loading the binary format. One way to describe the problem is that qpy the Python library is developed as part of terra and is tested for compatibility within terra and for compatibility with loading previous versions of the binary format, but these tests are always of the qpy Python library integrated into a version of terra. With qiskit-ibm-runtime and qiskit-ibm-provider, we have the qpy Python library effectively split out as a separate package from terra, and that opens up the question of compatibility between the qpy library version 0.22 and terra version 0.23 for example. (My prevous suggestion was to test terra 0.23 with the 0.22 qpy library somehow).

The solution to me is just for Runtime to update their vendored version of QPY a little faster.

I think this is almost right. What I think could mostly solve the issue is if on the server the runtime used terra's qpy library to load the payload rather than the vendored runtime one. I think the purpose of the vendored qpy is to prevent a user updating terra before the runtime service's version of terra is updated and generating a qpy payload with a qpy version number higher than the service can load. So the runtime package holds back the version of qpy until the version of terra in the runtime service has been updated (that's what I interpret from https://github.com/Qiskit/qiskit-ibm-runtime/pull/577#issuecomment-1279252515). Since new versions of the qpy library should be able to load old versions of the binary format, I don't think the version of the server needs to be held back.
Using the terra qpy on the server still leaves the possibility that the client could update terra and find an incompatibility between the new terra and the old qpy library for serializing a payload, but at least in this case the user might just accept that even though a new terra has been released they need to wait for the server to support the new feature they tried to use. I could imagine a system where terra's qpy supports generating payloads in older qpy versions and then the runtime queries which version of qpy the server supports and generates a payload of that version with terra rather than using a vendored qpy. That might help avoid issues with incompatibilities due to code refactoring between the vendored qpy and the new terra but could not handle the case where you want to use a feature that only received support in the new version of qpy (and would probably be a pain to maintain -- distinguishing between code changes that can be adapted back to the previous version of qpy and those that are new and require the new format could be difficult).

Runtime started out using terra qpy for both server and client sides, but we had a number of cases where the server could not be updated in time (usually due to deployment issues). This caused jobs serialized with new qpy to fail. As a result we switched to using vendored qpy.
Like @jakelishman mentioned, runtime really should update its qpy more quickly. In this particular case, the update was delayed to avoid potential disruption to Summit work (and kind of fell through the cracks after).
@wshanks made a good point that runtime server doesn't need to use vendored qpy, since qpy is backward compatible and terra qpy will always be ahead of vendored qpy. This would make the server upgrade easier and thus speed up the process.

I could imagine a system where terra's qpy supports generating payloads in older qpy versions ... rather than using a vendored qpy

This would make it easier for runtime, but I'd image it'd a bit of work for terra.

bump

this will be fixed by #577

Since I'm here again (I'd completely forgotten about this): I think I get what you're saying a bit better now Will.  If I've got it right, what you're suggesting doesn't come to arbitrary forwards dependence, but it still ends up in us trying to get old versions of QPY to be able to load some new objects or objects whose properties have changed slightly, which we usually can't do without updating QPY code.
If that's the case, then I don't think a separate package would be much different from the current situation. We'd still be dependent on Runtime upgrading a dependency that they'd presumably have pinned, which is very similar to updating a vendored version.

fixed by #577


feat(tutorial): complete step 2

Closes #
{{short description}}
Changelog
New

{{new thing}}

Changed

{{change thing}}

Removed

{{removed thing}}

New pull request from branches


I can't find XivAlexanderLoader.exe , XivAlexander.dll , or XivAlexander.dll.json

I downloaded the zip file XivAlexander-main and its contents don't include any of the above files.
I don't know if i need another program to run XivAlexander.
Any insight would be appreciated.

Try using the Releases page. Also, readme file has been updated, to prevent people from downloading the source code instead of the program.


Security Improvements

After a security review meeting with Larry Joy (Microsoft) some recommendations were made:
Recommended as ‘best practices’ when doing crypto in the browser:
Use the Web Crypto API when available:
Pros:
Crypto is performed in this OS and not in browser/JavaScript-engine. The keys and intermediate computations cannot be accessed through the browser dev tools.
No need for third-party packages
Crypto code maintained by OS
Better performance
Cons:
Can be more difficult to program
Not available on all platforms
May not have specific required algorithms
Don’t use AES-CBC if possible.  It’s not broken per se - but it has weaknesses and is not an authenticated form of encryption.
As discussed, using CBC here in offline mode doesn’t appear to be risk.
Suggestion:
Combine the two-step authentication (hash validation then decrypt validation) into a single step (password directly to encryption key and attempt validation decryption).
The two step works fine, but theoretically provides a larger attack surface to figure out the password.
We discussed a possible password-change scenario where the two-step method could make sense, but we couldn’t recall the details during the meeting. Something to consider.
Scenarios we discussed that need to be explored:
Could a user take an offline mode tablet and gain access to the server when it returns online?
While no riskier than taking an ‘online’ device, offline mode authentication may not timeout the same way online mode does, and any offline timeout may be circumvented with the dev tools
When returning to online mode, an expired session cookie ‘should’ prevent the server connection.  Testing should be done to make sure this is the case, and that the cookie is not auto-renewed.
What are the security/usability risks when the app rapidly moves in and out of offline/online mode?
This could be a mess and require some testing – or making online/offline mode switches manual.
Things I considered later:
If a user Alice, in offline mode, enters some data that is encrypted – then user Bob, on the same device, also in offline mode, logs in enters additional encrypted data, can the app data store handle data encrypted with multiple user keys?
If two offline devices store conflicting data about ‘Joe Smith’, what data ‘wins’ when these devices return online and the data is sent to the server?

Example code for working with Web Crypto API
(async () => {
    const username = "User1";
    const password = "k8FD61mNuh9M";
    const salt = crypto.getRandomValues(new Uint8Array(16));

    const data = {
        user: "Joe Smith",
        data: 123,
    };

    // creates AES-GCM key from a username/password
    const key = await passwordToAESKey(`${username}${password}`, salt);

    // encrypt text with the key
    const encrypted = await encrypt(JSON.stringify(data), key);

    // decrypts back to the original text
    const plain = await decrypt(encrypted, key)

    console.log(plain);  // should be the 'data' as JSON
})()

async function passwordToAESKey(password, salt) {
    const encoder = new TextEncoder();
    const utf8Bytes = encoder.encode(password);

    // import the password as a PBKDF2 key
    const pbkdf2Key = await crypto.subtle
        .importKey("raw", utf8Bytes, { name: "PBKDF2" },
            false, // not exportable (the key data cannot be exported)
            ["deriveKey"]
        )
        .catch((err) => {
            throw err;
        });

    // derive a new crypto key from the pbkdf2 key
    const aesKey = await crypto.subtle
        .deriveKey(
            {
                name: "PBKDF2",
                salt: salt,
                iterations: 2000,  // slows the password hashing down like in bcrypt
                hash: { name: "SHA-256" },
            },
            pbkdf2Key,
            { name: "AES-GCM", length: 256, },
            false,
            ["encrypt", "decrypt"]
        )
        .catch((err) => {
            throw err;
        });

    // this key object is opaque to the JavaScript environment
    // the actual key material and crypto computations using it will be in the OS and not in JavaScript
    return aesKey;
}

async function encrypt(text, key, additionalData = new Uint8Array()) {
    const encoder = new TextEncoder();
    const utf8Bytes = encoder.encode(text);

    // generate a new random IV - we'll need this for decryption
    const iv = crypto.getRandomValues(new Uint8Array(12));
    const cipherBytes = await crypto.subtle
        .encrypt(
            {
                name: "AES-GCM",
                // always generate a new iv every time you encrypt with a given key
                // recommended to use 12 bytes
                iv,
                additionalData,  // optional (if you do provide this, you'll also need it for decryption)
                tagLength: 128, // 128 (default)
            },
            key,
            utf8Bytes
        )
        .catch(function (err) {
            throw err;
        });

    // prepend the iv to the cipher since we'll need this IV to decrypt
    return new Uint8Array([...iv, ...new Uint8Array(cipherBytes)]);
}

async function decrypt(cipher, key, additionalData = new Uint8Array()) {
    let utf8decoder = new TextDecoder();
    // iv is the first 12 bytes - the remainder is the cipher
    const iv = cipher.slice(0, 12);
    cipher = cipher.slice(12);
    const plainBytes = await crypto.subtle
        .decrypt(
            {
                name: "AES-GCM",
                iv,
                additionalData, // only required if you provided this at encryption
                tagLength: 128,
            },
            key,
            cipher
        )
        .catch(function (err) {
            console.log(err);
            return new Uint8Array();
        });

    return utf8decoder.decode(plainBytes);
}


MINOR; Replace gt and lt char with html encoding

Use > and < instead of > and < to properly render the characters in HTML.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Thanks @chia7712 . I fixed those encoding issues.

https://github.com/apache/kafka/blob/trunk/docs/ops.html#L4249

I couldn't find anything wrong with that line. Did you mean to link to another line? Or did I miss something?

I couldn't find anything wrong with that line. Did you mean to link to another line? Or did I miss something?

true -> true

Thanks @chia7712 . Done.


Add unit tests

As discussed in #5, we need to decide how to approach testing.
There are at least two testing frameworks for GDScript. Tests can be implemented in GDScript within a Godot project using a testing framework addon. Such tests could be placed in the demo project or in a separate project dedicated to testing. The main advantage of this approach is that it executes code via GDScript - just like users would - and provides API usage examples. This setup makes it easier to verify if we're breaking the public API.
Another option is to use one of the C++ testing frameworks. Godot Engine already uses the C++ doctest framework, and any Godot module can add its own tests. If we add an option to build Sentry SDK as a Godot module, we could also add tests using this framework. Example: https://github.com/limbonaut/limboai/tree/master/tests. The advantages of this approach include no extra dependencies, and the ability to test internal things that are not part of the public API. However, sentry-godot can't currently be compiled as an engine module, and it would require some work to port. Such tests are compiled with the engine (using a specific build option) and can be executed via a CLI option, like so: godot --test --tc="*[SOME_TEST_GROUP]*"

Needs #16 for CI integration


Manifests using generateName will only keep last of each kind

Environmental Info:
K3s Version: v1.26.3+k3s1

Node(s) CPU architecture, OS, and Version:

Linux rpi3bp02 6.1.19-v8+ #1637 SMP PREEMPT Tue Mar 14 11:11:47 GMT 2023 aarch64 GNU/Linux
Cluster Configuration:

1 server, 3 agents
Describe the bug:

When placing a manifest in /var/lib/rancher/k3s/server/manifests with more than one instance of a resource kind that uses generateName, only the last one is actually created in the cluster.
Steps To Reproduce:

Installed K3s:
Place the attached manifest in /var/lib/rancher/k3s/server/manifests

Expected behavior:

I expected two Secrets to appear in the default namespace, one named runner-token-<something>, and another named batch-token-<something>.
Actual behavior:

Only the last Secret is created.
Additional context / logs:
manifests.yaml.txt

As discussed on slack I'm not sure this is easily fixable as wrangler's desiredset apply logic uses the object namespace and name as a key; for objects with no name but generateName the name portion of the key will just be an empty string, which will cause the controller to only store the last specified resource. We may need to just document it as a limitation.
https://github.com/rancher/wrangler/blob/master/pkg/apply/desiredset.go#L86-L92

Also I'm not sure quite how Kubernetes handles GenerateName but I wonder if the deploy controller might keep continually re-creating the same object over and over every time it's applied. We might need to explicitly guard against use of GenerateName.

@caroline-suse-rancher I am not even sure this belongs here, as the issue is in the rancher/wrangler package, not k3s.


Not an issue, but applause!

Not an issue, but applause!
A very practical job!

Not an issue, but applause! A very practical job!

thanks for your attention!


"converting" state

depending on size of svg, we may need to have a "converting" state where spinner / button states are triggered.

not needed now


feat: resolve.fallback and resolve.alias work with node: prefix

https://github.com/webpack/webpack/issues/14166

This is an Rspack feature, it has to handle the colon : first.


teleport_16: 16.0.4 -> 16.1.4

Description of changes
New minor version of Teleport 16. Upstream switched from Yarn to PNPM in both master and the v16 branch, so the derivation needed some changes.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 336353 run on x86_64-linux 1

6 packages built:

teleport (teleport_16)
teleport.client (teleport_16.client)
teleport_14
teleport_14.client
teleport_15
teleport_15.client


Contributing Guidelines

Contributing Guidelines
Description
A contributing guidelines file has been created and the majority of information has been added. This task is not yet complete, so I will not request any reviews until I consider it finished. In the meantime, I will push commits as I work on the rest of it and I welcome comments at any time in the process.
Type of Change

[ ] Feature
[ ] Bug fix
[x] Documentation

Contributors

@antlet

Closes: #22

Great work so far! I think this is super helpful for others planning to help with this project. Please take a look at the feedback I provided and make sure to update your branch with main, since it looks like it's out of date.


wrong value of semimajor axis a shows

for example:
after clicking Solar System for ZTF24aajqycq
SsODNet - ssoCard
Name: 2000 YZ / 22180
https://fink-portal.org/ZTF24aajqycq
Dynamical parameters
a (AU): 3.4569542840424936e-08

Thank you @CharlesBell for reporting! I thought this was fixed in a previous PR (https://github.com/astrolabsoftware/fink-science-portal/pull/602), but apparently there are still some failures... I will inspect why the correction , and provide a correction.

Oh I see -- in SSODNET, they recently changed back their units for distances (https://ssp.imcce.fr/webservices/ssodnet/changelog):

version 1.1.8: Distance of unit changed from au to km (and we updated Fink accordingly at that time)
version 1.1.9: Reset the distance unit to au to meet user expectations.
I have to update the code in Fink accordingly then. Thank you for spotting it!

I love Fink
Thank you.

Thank you! Actually we identified that the problem is coming from a dependency of Fink. This is being resolved, and I will let you know when the service will be updated with the correct display :-)

Hi @CharlesBell -- the problem has been fixed (configuration file in Miriade not updated), and semi major axis are properly displayed in AU. Thanks again for reporting!


document DEBUG_MAX_ARRAY_LENGTH

This feature exists, but not documented.

Coverage increased (+1.2%) to 75.0% when pulling 0a069cbc37f383e0fa581aad81ebb7c15ec7c0d1 on darky:master into 22f993216dcdcee07eb0601ea71a917e4925a30a on visionmedia:master.

Coverage increased (+1.2%) to 75.0% when pulling 0a069cbc37f383e0fa581aad81ebb7c15ec7c0d1 on darky:master into 22f993216dcdcee07eb0601ea71a917e4925a30a on visionmedia:master.

The Note: underneath that table attempts to document it, but apparently doesn't do a very good job.

Yep, I just noticed, that here link to node.js doc about additional options.
But discouraged situation, that some options described in table.
Better to describe all node.js options or remove DEBUG_COLORS, DEBUG_DEPTH, DEBUG_SHOW_HIDDEN, because it described in node.js doc


adds exercise 1

@eddiejaoude using this repo to have records of learning exercises with Ruby & be useful revision docs


issue-145 - User Story page is created

PR Description
This Pull request is about fixed issue #145

To fix the issue, What I changed are:

Created a stories module to display user stories.

two user stories per row.(Tried to create 3 in a row, but it becomes so small and hard to read)

Added path in the router to let users can access the page through the menu bar
Created userData.js to help test the User Story Page.

How to get to the page

Open index.html locally.
Click User story in the menu bar

Result - User Story Page

Hi @redxzeta I fixed responsive and PetCard.css based on your comment