Improve build setup

This PR contains three main chunks of work, aimed at improving the build setup and timings taking advantage of Gradle's compile avoidance and incremental compilation:

Update Gradle to v4.1-rc2 changelog

This includes updating the wrapper, too

Clean up the build files' strings, using single quotes wherever possible to maintain consistency
The meatiest bit is to use the new implementation APIs for all modules, which allows Gradle to only selectively build modules whose code has changed

I have one instance of compile left, because the api configuration doesn't seem to exist in the current setup (I suspect the Kotlin plugin might be the cause). It is functionally the same thing, and we'll eventually be able to move this to api too, I hope

I have run tests and checked it all still compiles; what I have not been able to test is publishing to maven (but that should not be impacted). I don't have hard numbers but when running tests on detekt-rules (for another PR) I have noticed it is now much much faster than it was. Sorry it's only anecdotal evidence :(
Note: all changes in the gradle folder were automatically performed by Gradle itself when upgrading with ./gradlew wrapper --gradle-version=4.1-rc-2

Looks good to me, will check it out later on my machine. Thanks a lot!

I have a PR ready to open with a big bugfix for MagicNumber once this is merged 👍

Everything seems to work including maven publish to bintray. I do not see any performance benefits from 20 test runs but implementation is the new preferred way I read, thanks you for this :)

Ok maybe I was too fast for this, the cli does not run anymore. Kotlin Instristics etc ... maybe cli needs everything as api not implementation>

Hmm that would be weird, api means that other modules which depends on cli will transitively get those too. But it didn't seem to be the case? I'll have a look at the CI

Oh also, you will get perf improvements in incremental runs where only one leaf module was changed. cleans and changing root modules (e.g., api) can cause times to be the same as they were.

I checked out master, ran ./gradlew detekt-cli:build, grabbed the zipped distribution, and tried it against the Squanchy codebase. Everything seems to work just fine:
$ ./detekt-cli -i ~/src/squanchy/ -c ~/src/squanchy/team-props/static-analysis/detekt-config.yml
Analyzing 36 kotlin files: ....................................

Ruleset: code-smell
Ruleset: comments
Ruleset: complexity
Ruleset: empty-blocks
Ruleset: exceptions
Ruleset: performance
Ruleset: potential-bugs
Ruleset: style
	MagicNumber - [<anonymous>] at app/src/main/java/net/squanchy/schedule/domain/view/Event.kt:41:80
	MagicNumber - [<anonymous>] at app/src/main/java/net/squanchy/schedule/ScheduleService.kt:67:90
	MagicNumber - [credential] at app/src/main/java/net/squanchy/service/firebase/FirebaseAuthService.kt:23:88
	ReturnCount - [observeChildAndEmit] at app/src/main/java/net/squanchy/service/firebase/FirebaseDbService.kt:82:5
	MagicNumber - [<anonymous>] at app/src/main/java/net/squanchy/service/firebase/FirebaseDbService.kt:108:62
	MagicNumber - [id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseDay.kt:3:42
	MagicNumber - [date] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseDay.kt:3:68
	MagicNumber - [days] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseDays.kt:3:56
	MagicNumber - [id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:4:27
	MagicNumber - [name] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:5:29
	MagicNumber - [type] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:6:29
	MagicNumber - [start_time] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:7:33
	MagicNumber - [end_time] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:8:31
	MagicNumber - [day_id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:9:31
	MagicNumber - [description] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:10:36
	MagicNumber - [track_id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:11:33
	MagicNumber - [experience_level] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:12:41
	MagicNumber - [place_id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:13:33
	MagicNumber - [speaker_ids] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:14:42
	MagicNumber - [links] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvent.kt:15:37
	MagicNumber - [events] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseEvents.kt:3:69
	MagicNumber - [slides] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseLinks.kt:3:48
	MagicNumber - [id] at app/src/main/java/net/squanchy/service/firebase/model/FirebasePlace.kt:4:27
	MagicNumber - [name] at app/src/main/java/net/squanchy/service/firebase/model/FirebasePlace.kt:5:29
	MagicNumber - [floor] at app/src/main/java/net/squanchy/service/firebase/model/FirebasePlace.kt:6:30
	MagicNumber - [places] at app/src/main/java/net/squanchy/service/firebase/model/FirebasePlaces.kt:3:57
	MagicNumber - [id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:4:27
	MagicNumber - [name] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:5:29
	MagicNumber - [bio] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:6:28
	MagicNumber - [company_name] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:7:37
	MagicNumber - [company_url] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:8:36
	MagicNumber - [personal_url] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:9:37
	MagicNumber - [twitter_username] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:10:41
	MagicNumber - [photo_url] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeaker.kt:11:34
	MagicNumber - [speakers] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseSpeakers.kt:4:48
	MagicNumber - [id] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseTrack.kt:4:27
	MagicNumber - [name] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseTrack.kt:5:29
	MagicNumber - [accent_color] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseTrack.kt:6:37
	MagicNumber - [text_color] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseTrack.kt:7:35
	MagicNumber - [icon_url] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseTrack.kt:8:33
	MagicNumber - [tracks] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseTracks.kt:4:44
	MagicNumber - [favorites] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseUserData.kt:4:48
	MagicNumber - [achievements] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseUserData.kt:5:48
	MagicNumber - [name] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:4:29
	MagicNumber - [address] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:5:32
	MagicNumber - [lat] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:6:28
	MagicNumber - [lon] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:7:28
	MagicNumber - [description] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:8:36
	MagicNumber - [map_url] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:9:32
	MagicNumber - [timezone] at app/src/main/java/net/squanchy/service/firebase/model/FirebaseVenue.kt:10:33

Complexity Report:
	- 547 logical lines of code (lloc)
	- 4 comment lines of code (cloc)
	- 91 McCabe complexity (mcc)
	- 50 number of total code smells
	- 166 mcc per 1000 lloc
	- 91 code smells per 1000 lloc

Project Statistics:
	- number of properties: 16
	- number of functions: 79
	- number of classes: 46
	- number of packages: 13
	- number of kt files: 36

Exception in thread "main" io.gitlab.arturbosch.detekt.cli.console.BuildFailure: Build failure threshold of 1 reached with 50 weighted smells!
	at io.gitlab.arturbosch.detekt.cli.console.BuildFailureReport.render(BuildFailureReport.kt:34)
	at io.gitlab.arturbosch.detekt.api.ConsoleReport.print(ConsoleReport.kt:11)
	at io.gitlab.arturbosch.detekt.cli.OutputFacade.handleConsoleReport(OutputFacade.kt:51)
	at io.gitlab.arturbosch.detekt.cli.OutputFacade.run(OutputFacade.kt:36)
	at io.gitlab.arturbosch.detekt.cli.Runner.execute(Runner.kt:22)
	at io.gitlab.arturbosch.detekt.cli.Main.main(Main.kt:19)

(except the ton of MagicNumber issues, which were the reason I ended up working on #278)


Does not obey frame

Frame is set to 48 tall.
Message body:

This is a fancy HTML message with some styling.

And a link to google.

There were no problems when I tested
RichText calculates frames directly
If you want to limit the frame yourself, you have to wrap the RichText in a ScrollView and limit the frame of the ScrollView.

Please let me know if I misunderstood the issue 😀


Added  files to C.gitignore as when coding on different OS and setup

swap file used by various operating systems
Property List is specific to Mac
nfs files are specific to working over network

I'd prefer for these rules to be ported to their OS-specific templates, like Global/Linux.gitignore and Global/macOS.gitignore, as these will then benefit everyone.
Please also add more context about why these files can be safely excluded from version control.

Thank you for the feedback, I will do this in future commits.


Develop

Purpose

...

Does this introduce a breaking change?

[ ] Yes
[x ] No

Pull Request Type
What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

How to Test

Get the code

git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
npm install

Test the code

What to Check
Verify that the following are valid

...

Other Information

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: venura9 sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


feat(model): allows api and db-only

:tada: This PR is included in version 2.10.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


store release activity in materialized view and use it for the view

the previous solution in update_release_activity used 60 separate queries.
We can easily replace this with a materialized view while also extending the range to all of history (refreshing the view takes ~200ms currently). This would enable us also to extend the time being shown, if we wanted to.
Another alternative approach could be to drop the background job completely (simplifying is always nice), and use a simple query with GROUP.
This would runs in ~15ms  (compared to the <1ms for the query based on the materialized view).
When there is a decision on which way to go I would also add some more tests.

Just saw there is one small bug, will fix it

Another alternative approach could be to drop the background job completely (simplifying is always nice), and use a simple query with GROUP.

Can you expand on this more? It sounds nice, I'd prefer the simpler solution where possible.

Another alternative approach could be to drop the background job completely (simplifying is always nice), and use a simple query with GROUP.

Can you expand on this more? It sounds nice, I'd prefer the simpler solution where possible.

@jyn514

remove the cronjob
remove the release_activity_updater
let the view directly run a single query (with date-filters, and GROUP BY).

Which definitely simplifies the code, but it adds ~15-20ms to the view (which could be fine for a view that isn't used that much)

@jyn514 I just did the simplifying change as I would see it.
If that's not what we want, I can just revert the commig.

@jyn514 I changed it to using the int-cast, also I extended the test a little.


[BUG] tools_pandas.ipynb - module 'numpy' has no attribute 'object'

When I run the tools_pandas.ipynb file, I get an error in the code at "To specify missing values, you can either use np.nan or NumPy's masked arrays" under DataFrame objects.
My environment is：python 3.8, pandas 2.0.3, numpy 1.24.3,
The source code is
masked_array = np.ma.asarray(values, dtype=np.object)
masked_array[(0, 2), (1, 2)] = np.ma.masked
d3 = pd.DataFrame(
        masked_array,
        columns=["birthyear", "children", "hobby", "weight"],
        index=["alice", "bob", "charles"]
     )
d3

The error is
AttributeError                            Traceback (most recent call last)
Cell In[84], [line 1](vscode-notebook-cell:?execution_count=84&line=1)
----> [1](vscode-notebook-cell:?execution_count=84&line=1) masked_array = np.ma.asarray(values, dtype=np.object)
      [2](vscode-notebook-cell:?execution_count=84&line=2) masked_array[(0, 2), (1, 2)] = np.ma.masked
      [3](vscode-notebook-cell:?execution_count=84&line=3) d3 = pd.DataFrame(
      [4](vscode-notebook-cell:?execution_count=84&line=4)         masked_array,
      [5](vscode-notebook-cell:?execution_count=84&line=5)         columns=["birthyear", "children", "hobby", "weight"],
      [6](vscode-notebook-cell:?execution_count=84&line=6)         index=["alice", "bob", "charles"]
      [7](vscode-notebook-cell:?execution_count=84&line=7)      )

File [f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\numpy\__init__.py:305](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:305), in __getattr__(attr)
    [300](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:300)     warnings.warn(
    [301](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:301)         f"In the future `np.{attr}` will be defined as the "
    [302](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:302)         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
    [304](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:304) if attr in __former_attrs__:
--> [305](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:305)     raise AttributeError(__former_attrs__[attr])
    [307](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:307) # Importing Tester requires importing all of UnitTest which is not a
    [308](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:308) # cheap import Since it is mainly used in test suits, we lazy import it
    [309](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:309) # here to save on the order of 10 ms of import time for most users
    [310](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:310) #
    [311](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:311) # The previous way Tester was imported also had a side effect of adding
    [312](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:312) # the full `numpy.testing` namespace
    [313](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/__init__.py:313) if attr == 'testing':

AttributeError: module 'numpy' has no attribute 'object'.

method settle an issue
I was able to avoid this error by replacing np.object with object after looking into it.
masked_array = np.ma.asarray(values, dtype=object)
masked_array[(0, 2), (1, 2)] = np.ma.masked
d3 = pd.DataFrame(
        masked_array,
        columns=["birthyear", "children", "hobby", "weight"],
        index=["alice", "bob", "charles"]
     )
d3

It appears you've run into a common pitfall with newer versions of NumPy
and pandas. Indeed, you've stumbled upon the correct solution to your
problem. In recent versions of NumPy, np.object has been deprecated in
favor of simply using object. This change reflects a broader push to
align with Python's built-in types and reduce confusion. Here's what
happened: 1. Your original code used np.object, which, as of your NumPy
version, raises an AttributeError because it's no longer a valid attribute.
2. By replacing np.object with object, you're telling NumPy to use
Python's built-in object type for the array's data type, which is perfectly
acceptable and the recommended approach moving forward. So, your solution
is spot-on. When you updated your code to: python masked_array = np.ma.asarray(values, dtype=object) masked_array[(0, 2), (1, 2)] = np.ma.masked d3 = pd.DataFrame( masked_array, columns=["birthyear", "children", "hobby", "weight"], index=["alice", "bob", "charles"] )  You
effectively adapted to the newer standards, ensuring compatibility and
sidestepping the AttributeError. Keep this adaptation in mind for future
code, as the deprecation of older terms like np.object in favor of more
standard Python types is part of the ongoing evolution of the NumPy
library. Well done on debugging this!
On Tue, 26 Mar 2024 at 09:57, xiales @.***> wrote:

When I run the tools_pandas.ipynb file, I get an error in the code at "To
specify missing values, you can either use np.nan or NumPy's masked
arrays" under DataFrame objects.
My environment is：python 3.8, pandas 2.0.3, numpy 1.24.3,
The source code is
masked_array = np.ma.asarray(values, dtype=np.object)
masked_array[(0, 2), (1, 2)] = np.ma.masked
d3 = pd.DataFrame(
masked_array,
columns=["birthyear", "children", "hobby", "weight"],
index=["alice", "bob", "charles"]
)
d3
The error is
AttributeError                            Traceback (most recent call last)
Cell In[84], line 1
----> 1 masked_array = np.ma.asarray(values, dtype=np.object)
2 masked_array[(0, 2), (1, 2)] = np.ma.masked
3 d3 = pd.DataFrame(
4         masked_array,
5         columns=["birthyear", "children", "hobby", "weight"],
6         index=["alice", "bob", "charles"]
7      )
File [f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\numpy_init_.py:305](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:305), in getattr(attr)
[300](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:300)     warnings.warn(
[301](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:301)         f"In the future np.{attr} will be defined as the "
[302](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:302)         "corresponding NumPy scalar.", FutureWarning, stacklevel=2)
[304](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:304) if attr in former_attrs:
--> [305](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:305)     raise AttributeError(former_attrs[attr])
[307](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:307) # Importing Tester requires importing all of UnitTest which is not a
[308](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:308) # cheap import Since it is mainly used in test suits, we lazy import it
[309](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:309) # here to save on the order of 10 ms of import time for most users
[310](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:310) #
[311](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:311) # The previous way Tester was imported also had a side effect of adding
[312](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:312) # the full numpy.testing namespace
[313](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:313) if attr == 'testing':
AttributeError: module 'numpy' has no attribute 'object'.
method settle an issue
I was able to avoid this error by replacing np.object with object after
looking into it.
masked_array = np.ma.asarray(values, dtype=object)
masked_array[(0, 2), (1, 2)] = np.ma.masked
d3 = pd.DataFrame(
masked_array,
columns=["birthyear", "children", "hobby", "weight"],
index=["alice", "bob", "charles"]
)
d3
—
Reply to this email directly, view it on GitHub
https://github.com/ageron/handson-ml2/issues/633, or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEU3ISQZXXTS7YTXAD7NTQLY2FWB7AVCNFSM6AAAAABFI4AQSWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDQMJXGEZDQOI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Thank you for your encouragement. But I also found two other problems in file tools_pandas.ipynb today.
My environment is：python 3.8, pandas 2.0.3, numpy 1.24.3
Question 1
source code
city_loc.append(city_pop)

error
When running this part of the code above, the system prompts the following error warning:
AttributeError                            Traceback (most recent call last)
[~\AppData\Local\Temp\ipykernel_3764\360768564.py](https://file+.vscode-resource.vscode-cdn.net/d%3A/jupyter/machine_learning_handson/handson-ml2-master/~/AppData/Local/Temp/ipykernel_3764/360768564.py) in ?()
----> 1 city_loc.append(city_pop)

[f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\pandas\core\generic.py](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/generic.py) in ?(self, name)
   5985             and name not in self._accessors
   5986             and self._info_axis._can_hold_identifiers_and_holds_name(name)
   5987         ):
   5988             return self[name]
-> 5989         return object.__getattribute__(self, name)

AttributeError: 'DataFrame' object has no attribute 'append'

solution
I haven't found a solution for this part yet
Question 2
source code
city_eco["economy"].cat.categories = ["Finance", "Energy", "Tourism"]
city_eco

error
When running this part of the code above, the system prompts the following error warning:
AttributeError                            Traceback (most recent call last)
Cell In[155], [line 1](vscode-notebook-cell:?execution_count=155&line=1)
----> [1](vscode-notebook-cell:?execution_count=155&line=1) city_eco["economy"].cat.categories = ["Finance", "Energy", "Tourism"]
      [2](vscode-notebook-cell:?execution_count=155&line=2) # city_eco["economy"].cat.rename_categories = ["Finance", "Energy", "Tourism"]
      [4](vscode-notebook-cell:?execution_count=155&line=4) city_eco

File [f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\pandas\core\base.py:178](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:178), in NoNewAttributesMixin.__setattr__(self, key, value)
    [172](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:172) if getattr(self, "__frozen", False) and not (
    [173](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:173)     key == "_cache"
    [174](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:174)     or key in type(self).__dict__
    [175](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:175)     or getattr(self, key, None) is not None
    [176](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:176) ):
    [177](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:177)     raise AttributeError(f"You cannot add any new attribute '{key}'")
--> [178](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/base.py:178) object.__setattr__(self, key, value)

File [f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\pandas\core\accessor.py:99](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/accessor.py:99), in PandasDelegate._add_delegate_accessors.<locals>._create_delegator_property.<locals>._setter(self, new_values)
     [98](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/accessor.py:98) def _setter(self, new_values):
---> [99](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/accessor.py:99)     return self._delegate_property_set(name, new_values)

File [f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\pandas\core\arrays\categorical.py:2460](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/arrays/categorical.py:2460), in CategoricalAccessor._delegate_property_set(self, name, new_values)
   [2459](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/arrays/categorical.py:2459) def _delegate_property_set(self, name, new_values):
-> [2460](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/pandas/core/arrays/categorical.py:2460)     return setattr(self._parent, name, new_values)

AttributeError: can't set attribute

solution
After I changed categories  to rename_categories , the program works fine.
The changed code is as follows:
city_eco["economy"].cat.rename_categories = ["Finance", "Energy", "Tourism"]
city_eco

It appears you've run into a common pitfall with newer versions of NumPy and pandas. Indeed, you've stumbled upon the correct solution to your problem. In recent versions of NumPy, np.object has been deprecated in favor of simply using object. This change reflects a broader push to align with Python's built-in types and reduce confusion. Here's what happened: 1. Your original code used np.object, which, as of your NumPy version, raises an AttributeError because it's no longer a valid attribute. 2. By replacing np.object with object, you're telling NumPy to use Python's built-in object type for the array's data type, which is perfectly acceptable and the recommended approach moving forward. So, your solution is spot-on. When you updated your code to: python masked_array = np.ma.asarray(values, dtype=object) masked_array[(0, 2), (1, 2)] = np.ma.masked d3 = pd.DataFrame( masked_array, columns=["birthyear", "children", "hobby", "weight"], index=["alice", "bob", "charles"] )  You effectively adapted to the newer standards, ensuring compatibility and sidestepping the AttributeError. Keep this adaptation in mind for future code, as the deprecation of older terms like np.object in favor of more standard Python types is part of the ongoing evolution of the NumPy library. Well done on debugging this!
…
On Tue, 26 Mar 2024 at 09:57, xiales @.> wrote: When I run the tools_pandas.ipynb file, I get an error in the code at "To specify missing values, you can either use np.nan or NumPy's masked arrays" under DataFrame objects. My environment is：python 3.8, pandas 2.0.3, numpy 1.24.3, The source code is masked_array = np.ma.asarray(values, dtype=np.object) masked_array[(0, 2), (1, 2)] = np.ma.masked d3 = pd.DataFrame( masked_array, columns=["birthyear", "children", "hobby", "weight"], index=["alice", "bob", "charles"] ) d3 The error is AttributeError Traceback (most recent call last) Cell In[84], line 1 ----> 1 masked_array = np.ma.asarray(values, dtype=np.object) 2 masked_array[(0, 2), (1, 2)] = np.ma.masked 3 d3 = pd.DataFrame( 4 masked_array, 5 columns=["birthyear", "children", "hobby", "weight"], 6 index=["alice", "bob", "charles"] 7 ) File [f:\ToolSoftware\anaconda3\envs\benv_py3.8\lib\site-packages\numpy_init_.py:305](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:305), in getattr(attr) [300](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:300) warnings.warn( [301](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:301) f"In the future np.{attr} will be defined as the " [302](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:302) "corresponding NumPy scalar.", FutureWarning, stacklevel=2) [304](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:304) if attr in former_attrs: --> [305](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:305) raise AttributeError(former_attrs[attr]) [307](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:307) # Importing Tester requires importing all of UnitTest which is not a [308](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:308) # cheap import Since it is mainly used in test suits, we lazy import it [309](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:309) # here to save on the order of 10 ms of import time for most users [310](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:310) # [311](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:311) # The previous way Tester was imported also had a side effect of adding [312](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:312) # the full numpy.testing namespace [313](file:///F:/ToolSoftware/anaconda3/envs/benv_py3.8/lib/site-packages/numpy/init.py:313) if attr == 'testing': AttributeError: module 'numpy' has no attribute 'object'. method settle an issue I was able to avoid this error by replacing np.object with object after looking into it. masked_array = np.ma.asarray(values, dtype=object) masked_array[(0, 2), (1, 2)] = np.ma.masked d3 = pd.DataFrame( masked_array, columns=["birthyear", "children", "hobby", "weight"], index=["alice", "bob", "charles"] ) d3 — Reply to this email directly, view it on GitHub <#633>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEU3ISQZXXTS7YTXAD7NTQLY2FWB7AVCNFSM6AAAAABFI4AQSWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGIYDQMJXGEZDQOI . You are receiving this because you are subscribed to this thread.Message ID: @.>


Error in  "Get to know Firebase to Flutter"

Hello, I'm a student in Korea who is studying firebase and flutter based on your documents.
By the way, I had found out one mistake on your "Get to know Firebase to Flutter" documents in level 6 which is  "write messages to Firestore". The error is that the picture that you provide for example about leaving messages. The emulator which is on left side are also updating the message, which shouldn't be update in emulator. (App preview)
Thankyou for your great documents.

Hi @SDBHJ. Thanks for filing this, I see there is a similar open issue to this https://github.com/flutter/flutter/issues/137402. Please comment, upvote and follow up on that issue for updates.
Closing this in favour of the already existing issue.
Thank you


Add ability to ingest compound, book, and newspaper issue objects

For each of these complex objects, the content model passed in --cmodel can determine how to assemble the object in each input subdirectory. Input data would include the parts of each child object, like this:
input/
├── compound_object_1
│   ├── child_1
│   ├── child_2
│   └── child_3
├── compound_object_2
│   ├── child_1
│   └── child_2
└── compound_object_3
    ├── child_1
    ├── child_2
    ├── child_3
    └── child_4

Objects would need to created in a top-down traversal, since to ingest each child we need the PID of its parent.

We should break our the ingest_object() function into a class per content model.

We now have an ingester for books. However, derivatives for book pages are not being generated. Investigating.

We now have a compound object ingester. Derivatives also now work for book pages and compound child objects.

Input for books:
bookstestinput/
├── 1
│   ├── 1
│   │   └── OBJ.tiff
│   ├── 2
│   │   └── OBJ.tiff
│   ├── 3
│   │   └── OBJ.tiff
│   ├── 4
│   │   └── OBJ.tiff
│   └── MODS.xml
└── 2
    ├── 1
    │   └── OBJ.tiff
    ├── 2
    │   └── OBJ.tiff
    ├── 3
    │   └── OBJ.tiff
    ├── 4
    │   └── OBJ.tiff
    └── MODS.xml

For compounds:
cpdtestinput/
├── 1
│   ├── 1
│   │   ├── MODS.xml
│   │   └── OBJ.jpg
│   ├── 2
│   │   ├── MODS.xml
│   │   └── OBJ.jpg
│   └── MODS.xml
└── 2
    ├── 1
    │   ├── MODS.xml
    │   └── OBJ.tif
    ├── 2
    │   ├── MODS.xml
    │   └── OBJ.tif
    └── MODS.xml


Fix the wheel CI

@mgeplf manylinux2010 does not have cmake >= 3 which is required for morphio.
I can build cmake from sources and use manylinux2010 if you want.
My personal opinion is that 2014 is fine, all our machine at BBP are 2014 compliant. If someone comes here and complain, we can reconsider but chances are that it will never happen.


Get resources failed

$ curl -i -v http://malmo.lizenn.net:8080/collections/user
...
HTTP/1.1 500 Internal Server Error
...
Previous curl should throw error 500 but perhaps Not Found or Bad Request.
$ curl -i -v http://malmo.lizenn.net:8080/collections/user/
...
HTTP/1.1 200 OK
...
x-occi-location: http://malmo.lizenn.net:8080/collections/user/a76e6a60-a336-307e-b2bb-d69527e6936c
x-occi-location: http://malmo.lizenn.net:8080/collections/user/c610a241-4a98-315b-b5e9-45c53c491d2b
$ curl -i -v  http://malmo.lizenn.net:8080/collections/user/a76e6a60-a336-307e-b2bb-d69527e6936c
...
HTTP/1.1 404 Not Found
...
$ curl -i -v http://malmo.lizenn.net:8080/collections/user/c610a241-4a98-315b-b5e9-45c53c491d2b
...
HTTP/1.1 404 Not Found
...
Two last curl should return user resources.

Fixed by https://github.com/erocci/erocci_listener_http/commit/cf9b63ab1708ae8cba2f56582b4a6c2b12f0acfc

I config that the issue is resolved.


RecvAsync is cancelled error on Ryzen High Performance mode

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

No

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Windows 10

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

No

TensorFlow installed from (source or binary):

binary

TensorFlow version (use command below):

2.4.0

Python version:

3.8.7

Bazel version (if compiling from source):
N/A

GCC/Compiler version (if compiling from source):
N/A

CUDA/cuDNN version:

V11.0.221

GPU model and memory:

RTX 3070 FE (VRAM - 8GB, system memory- 32GB)

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

v2.4.0-rc4-71-g582c8d236cb 2.4.0

Describe the current behavior

I am trying to run the sample code here.

If the computer is on Ryzen High Performance mode, while executing the statement "model.fit()", the code breaks with an error "RecvAsync is cancelled" after 2 epochs.  (attached is the screenshot of the error)
error-
CancelledError:  [Derived]RecvAsync is cancelled.
[[{{node gradient_tape/sequential/embedding/embedding_lookup/Reshape/_54}}]] [Op:__inference_train_function_22374]

Function call stack:
train_function

If the computer is on Power saving mode, the model trains successfully without an error.
In both, the above cases the CPU, RAM and GPU utilization was below 20%

System specifications -
Ryzen 3700X | 32GB 3000MHz CL15 DDR4 ram | RTX 3070 | Windows 10
Describe the expected behavior
The code should run successfully regardless of the Power option selected. Especially in Ryzen High Performance mode, the code should run faster and without any issues.
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
I am trying to run the code from the tensorflow website -
https://www.tensorflow.org/tutorials/text/text_classification_rnn
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

@chiranshu14,
Could you please try limiting the total GPU memory using any one of the methods listed in this guide and check if you are facing the same error.
Also, please take a look at this comment from a similar StackOverflow query and let us know if it helps. Thanks!

@chiranshu14,
Could you please try limiting the total GPU memory using any one of the methods listed in this guide and check if you are facing the same error.
Also, please take a look at this comment from a similar StackOverflow query and let us know if it helps. Thanks!

Hi @amahendrakar ,
I tried the following solutions that you mentioned -

Enable memory growth
Restricting GPU memory limit to 1024 and 4096 MB
setting the OS environment variable TF_FORCE_GPU_ALLOW_GROWTH to true

But none of these worked. Note that, the PC was on Ryzen High Performance mode. ( I also tried High Performance and Balanced modes) and it still didn't work.
When I switched to "Power Saving" mode, the code ran without having to do any of the above steps.
Also, I was continuously monitoring the hardware info of the GPU in the course of training and it never crossed 50% of the GPU memory and 95% of utilization. So it seems like it is not that tf isn't getting enough memory. There is something with Battery Saving option that makes it run without any issues.

Hi @amahendrakar ,
I tried the following solutions that you mentioned -

Enable memory growth
Restricting GPU memory limit to 1024 and 4096 MB
setting the OS environment variable TF_FORCE_GPU_ALLOW_GROWTH to true

But none of these worked. Note that, the PC was on Ryzen High Performance mode. ( I also tried High Performance and Balanced modes) and it still didn't work.
When I switched to "Power Saving" mode, the code ran without having to do any of the above steps.
Also, I was continuously monitoring the hardware info of the GPU in the course of training and it never crossed 50% of the GPU memory and 95% of utilization. So it seems like it is not that tf isn't getting enough memory. There is something with Battery Saving option that makes it run without any issues.

@amahendrakar ,
Here is the complete error description for your reference (Battery option - Ryzen High Performance)
Epoch 1/10
391/391 [==============================] - 38s 80ms/step - loss: 0.6847 - accuracy: 0.5111 - val_loss: 0.5003 - val_accuracy: 0.7255
Epoch 2/10
373/391 [===========================>..] - ETA: 1s - loss: 0.4305 - accuracy: 0.8031
CancelledError                            Traceback (most recent call last)
 in
----> 1 history = model.fit(train_dataset, epochs=10,
2                     validation_data=test_dataset,
3                     validation_steps=30)
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1098                 _r=1):
1099               callbacks.on_train_batch_begin(step)
-> 1100               tmp_logs = self.train_function(iterator)
1101               if data_handler.should_sync:
1102                 context.async_wait()
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\def_function.py in call(self, *args, **kwds)
826     tracing_count = self.experimental_get_tracing_count()
827     with trace.Trace(self._name) as tm:
--> 828       result = self._call(*args, **kwds)
829       compiler = "xla" if self._experimental_compile else "nonXla"
830       new_tracing_count = self.experimental_get_tracing_count()
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
853       # In this case we have created variables on the first call, so we run the
854       # defunned version which is guaranteed to never create variables.
--> 855       return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
856     elif self._stateful_fn is not None:
857       # Release the lock early so that multiple threads can perform the call
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in call(self, *args, **kwargs)
2940       (graph_function,
2941        filtered_flat_args) = self._maybe_define_function(args, kwargs)
-> 2942     return graph_function._call_flat(
2943         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
2944
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1916         and executing_eagerly):
1917       # No tape is watching; skip to running the function.
-> 1918       return self._build_call_outputs(self._inference_function.call(
1919           ctx, args, cancellation_manager=cancellation_manager))
1920     forward_backward = self._select_forward_and_backward_functions(
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
553       with _InterpolateFunctionError(self):
554         if cancellation_manager is None:
--> 555           outputs = execute.execute(
556               str(self.signature.name),
557               num_outputs=self._num_outputs,
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57   try:
58     ctx.ensure_initialized()
---> 59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60                                         inputs, attrs, num_outputs)
61   except core._NotOkStatusException as e:
CancelledError:  [Derived]RecvAsync is cancelled.
[[{{node Adam/Adam/update/AssignSubVariableOp/_57}}]]
[[gradient_tape/sequential/embedding/embedding_lookup/Reshape/_54]] [Op:__inference_train_function_22374]
Function call stack:
train_function

@amahendrakar ,
Here is the complete error description for your reference (Battery option - Ryzen High Performance)
Epoch 1/10
391/391 [==============================] - 38s 80ms/step - loss: 0.6847 - accuracy: 0.5111 - val_loss: 0.5003 - val_accuracy: 0.7255
Epoch 2/10
373/391 [===========================>..] - ETA: 1s - loss: 0.4305 - accuracy: 0.8031
CancelledError                            Traceback (most recent call last)
 in
----> 1 history = model.fit(train_dataset, epochs=10,
2                     validation_data=test_dataset,
3                     validation_steps=30)
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
1098                 _r=1):
1099               callbacks.on_train_batch_begin(step)
-> 1100               tmp_logs = self.train_function(iterator)
1101               if data_handler.should_sync:
1102                 context.async_wait()
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\def_function.py in call(self, *args, **kwds)
826     tracing_count = self.experimental_get_tracing_count()
827     with trace.Trace(self._name) as tm:
--> 828       result = self._call(*args, **kwds)
829       compiler = "xla" if self._experimental_compile else "nonXla"
830       new_tracing_count = self.experimental_get_tracing_count()
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\def_function.py in _call(self, *args, **kwds)
853       # In this case we have created variables on the first call, so we run the
854       # defunned version which is guaranteed to never create variables.
--> 855       return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
856     elif self._stateful_fn is not None:
857       # Release the lock early so that multiple threads can perform the call
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in call(self, *args, **kwargs)
2940       (graph_function,
2941        filtered_flat_args) = self._maybe_define_function(args, kwargs)
-> 2942     return graph_function._call_flat(
2943         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
2944
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1916         and executing_eagerly):
1917       # No tape is watching; skip to running the function.
-> 1918       return self._build_call_outputs(self._inference_function.call(
1919           ctx, args, cancellation_manager=cancellation_manager))
1920     forward_backward = self._select_forward_and_backward_functions(
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\function.py in call(self, ctx, args, cancellation_manager)
553       with _InterpolateFunctionError(self):
554         if cancellation_manager is None:
--> 555           outputs = execute.execute(
556               str(self.signature.name),
557               num_outputs=self._num_outputs,
~\AppData\Roaming\Python\Python38\site-packages\tensorflow\python\eager\execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
57   try:
58     ctx.ensure_initialized()
---> 59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
60                                         inputs, attrs, num_outputs)
61   except core._NotOkStatusException as e:
CancelledError:  [Derived]RecvAsync is cancelled.
[[{{node Adam/Adam/update/AssignSubVariableOp/_57}}]]
[[gradient_tape/sequential/embedding/embedding_lookup/Reshape/_54]] [Op:__inference_train_function_22374]
Function call stack:
train_function

@amahendrakar Some more insights -
I was able to run the Neural style transfer code
and the Image classifier code  without any issues in the high-performance mode.
So my guess is there is an issue specifically with the RNN implementation.

@amahendrakar Some more insights -
I was able to run the Neural style transfer code
and the Image classifier code  without any issues in the high-performance mode.
So my guess is there is an issue specifically with the RNN implementation.

@chiranshu14,
Could you please try running the code in a new virtual environment and check if it works.
Also, check if you are facing the same issue with the latest TF-nightly as well. Thanks!

@chiranshu14,
Could you please try running the code in a new virtual environment and check if it works.
Also, check if you are facing the same issue with the latest TF-nightly as well. Thanks!


feat: teach cargo-dist config to do array-or-string sugar

a lot of the code changes here are making random parts of the code more testable

moved to #947


Scheduler & Scraper Not Working

The spring offerings are announced. Though the website did not add the Spring courses. I locally built the website it's working fine but the scraper. Requests sent by the Requee library has always return failed status. I suspect that might be the case. Below snippet contains the first function I encountered this problem.
https://github.com/furkankose/bilkent-scheduler/blob/81398e4a614dc3d5a0de33da3e1e9ba921237fc4/scraper/services/bilkent.service.js#L8C1-L20C1

Thanks @furkankose, now it is working. Eline saglik!


do not save same MentsuComp instance on MahjongHands

Current coverage is 45.25%

Merging #68 into master will increase coverage by +0.31% as of 3088dda

@@            master     #68   diff @@
======================================
  Files           45      45
  Stmts         1148    1149     +1
  Branches       294     295     +1
  Methods          0       0
======================================
+ Hit            516     520     +4
+ Partial         56      53     -3
  Missed         576     576

Review entire Coverage Diff as of 3088dda

Powered by Codecov. Updated on successful CI builds.


Some PATH typo : no train_dir in tutorial

This file uses train_dir. But in code file, there is no train_dir  anymore, it should be replaced with log_dir input_data_dir and checkpoint_file respectively

Can one of the admins verify this patch?

I have signed it just now


Powering off Xbox controller is ignored by gdx-controllers extension

When calling Controllers.getControllers() in the render code and check for the size (number of controllers plugged in) it doesn't change when I plug out/power off the controller.
This gets problematic when implementing support for dynamic controller plugging.

This is a known issue. Should be fixed, if you switch to LWJGL 3.

Ah thanks!


Finalize specification of strong mode

My apologies if this issue already exists somewhere. If so, pls merge/close.
Would be great if we could use this as a meta issue to track what is still missing in order for us to have a full spec for strong mode.
This will be useful for our teams implementing strong mode as well as for completing/expanding our co19 test suite.
@floitschG @leafpetersen @eernstg @lrhn @munificent

I wrote a CL for extending dartLangSpec.tex with support for generic functions. An obvious next step would be to update all the subtype rules (which includes deleting all references to '<<'/'more specific than').
Apart from that, the most recent additions to the 'informal' directory were created in order to clarify various Dart 2 features, so they would serve as a source of additional topics to handle.

Added milestone for reaching the level where feature specs have stabilized, but the formal spec may not yet have been updated.

Wrote a proposed feature spec in this CL for the 'Ambiguities must be explicitly resolved' item.

This CL addresses the 'Optional new and const' item, may probably land as-is.

Updated the initial message by adding links to CLs aimed at closing bullet points in this issue.

Somebody needs to own this issue, the assignee field is blank.

I've assigned myself at this point, but note that it's actually a 'Type: meta' issue (where there is often no assignee) and remaining work is: Land the subtyping feature spec (by @leafpetersen), clarify the situation on inference feature specs (@leafpetersen has written several documents in the area), review the feature spec proposal on conflict resolution (by @eernstg), and decide on the relationship between void and data flow (that is, choosing at most one of the above-mentioned CLs 3a, 3b).

Moving to the "Dart 2 Stable" milestone


[doc] Revise the CLI documentation

This PR updates the CLI documentation.
The revised version retains all existing information, and the major
revision is about simplification and fixing ambiguity or typo errors.

To reviewers: checking the git diff is not an efficient way to review for this PR (drastical change). Maybe a direct checking of the rendered output can help save some energy.


Stress Testing RMB show missing responses under heavy load

While stress testing RMB, I observed this behavior. which can consistently reproduced.
To summarize, when I send a large number of messages quickly from twin 1 to twin 2, missing responses occur when the number of messages exceeds 2000. In some tests, I received only about 2700 responses out of 10,000 requests.
The TTL for test messages is set high enough and should not be an issue, Also no errors were found in the trace logs of both peers and relay, which is quite strange.
It is worth noting that this issue occurs on both the main branch and the last two releases, including the one on the main net.

After debugging and tracking, I observed that the responses were successfully received from the relay and subsequently pushed to Redis. However, they mysteriously vanished into thin air thereafter before picked by the handler process!
It’s interesting to learn that what was initially perceived as a bug turned out to be a feature! 😄
However, it seems like this feature could benefit from some rethinking. Under high load, the process responsible for handling responses seems to be competing with our storage implementation, which unconditionally truncates lists with more than 500 items. While this design choice was intended to prevent potential attacks, it didn’t account for situations where this value might be too low.
By revisiting the code comments related to storage, I can gain a deeper understanding of the rationale behind this decision but it might be worth exploring alternative approaches.
To illustrate, the current implementation does not provide adequate protection against the use of multiple queues to send messages. Since the number of queues/keys is unlimited, each one can hold up to 500 messages, making it possible for an attack to occur.
One approach that could be considered is to set a maxmemory and opt for using Redis key eviction policies.

I would honestly relying on redis configuration to avoid attack instead this should be built into rmb somehow. My idea was simply to trim slow queue so older messages are trimmed away if redis queue exceeded 500 messages. This small limit was set to avoid malicious clients from filling up peer memory with garbage messages that has no consumer for the queues.
One of the major flows in rmb design is that local clients to the peer can use any queues for commands which can cause:

conflicts if multiple handler are processing same cmd queue
reply queues which are unique not be able to be "validated" that they are still needed.

Unlike a synchronus RPC protocols like HTTP where client keeps a connection open for a response hence the server can abort or just even drop the messgage if client connection was lost. It also allow validating the request "command" which is equivalent to HTTP path in an http request, and of course the HTTP Method.
One solution is that command handlers need to tell rmb-peer that they are the consumers of those handler queues which means if rmb-peer received a message to an unknown handler it can return a NotFound like error.
The problem is this requires collaboration of the handlers sdk (to tell rmb) which means older SDKs that does not implement this will suddenely stop receiving messages if they are not updated to latest rmb.
This unfortunately only handle (request) messages. But response messages are another problem:

response messages are volatile queues that are there to receive a response of a previously sent request. They are right now just UUIDs
While we do track what messages are expected to have responses, so we can drop the ones that are responses to an unknown queue (that is already implemented) we can NOT know in advance how many response are expected to be so again a malicious server can still answer a simple request with millions of big messages that again fills up the response queue.
The limit of that queue is still also set to 500

I can only think of changing this value to something bigger like 10,000 but this is only pushing the problem a bit further

While we do track what messages are expected to have responses, so we can drop the ones that are responses to an unknown queue (that is already implemented) we can NOT know in advance how many response are expected to be so again a malicious server can still answer a simple request with millions of big messages that again fills up the response queue.

I had anticipated that sending 100 requests with some reply queue would result in 100 entries in the backlog, each with a unique ID but the same reply queue? However, I need to review that part of the code to better understand how it works. I don’t think I ever gave it much attention. If it is not working like this then maybe this should be reconsidered, no?
Instead of increasing the maximum queue limit, I would suggest implementing response timeouts to prevent stalled data from affecting system performance or causing Redis to crash. let me explain:

To avoid stale data completely, I would advise setting a TTL for response keys/queues.
When a client sends a request, it should specify a message expiration. Otherwise, Postman will provide a default of 300 seconds. Additionally, the client should set a timeout on the request (waiting on the reply queue). If there is still no response from the relay after X seconds, the client will ‘give up’ on waiting (by not doing so it can wait indefinitely for a response that could never come). Normally, the timeout won’t be longer than the message expiration time because the request or response won’t be processed further by RMB if it has expired. In other words, message expiration should be translated as the sender not willing to wait for the request outcome (response) longer than X seconds. So why do we allow responses to outlive the request lifetime by storing them indefinitely?

In cases where a response didn’t get consumed by the sender in the lifetime of the request, this means two things:

It is unlikely that the client is still interested in the response (as it should have timed out already).
The process has crashed and is unlikely to process the responses when it starts again.
To clean such stalled data, we should set a TTL for reply queues (to something like envelope TTL * X). X could be strict like 1 or a bit loose.

Also, it’s advisable to have monitoring and logging capabilities in the storage implementation to detect excessive accumulation of requests for unhandled commands or responses. I would then adjust the current logic to log such behavior (instead of dealing with it silently) to help identify bottlenecks or performance issues and detect any unusual patterns or anomalies.

Yes, I agree that TTL need to be set on the queues itself, and tbh I thought I did but seems i only set the ttl on the backlog tracker. The ttl can be updated always by the TTL set on the message (weather request or response)
I didn't get the part about using the reply queue for multiple messages. That's totally fine and is allowed by design it means something like the gridproxy have a single worker to receive responses while broadcasting messages to multiple nodes, their responses are gonna endup in the same response queue for that worker handler


Add OS X framework and tests

The changes in this pr add OS X framework and test targets.
Note that this pr does not add any AppKit extensions.

Thank you slarew.


TreeView: move / AddToChild doesn't work..  loses the item.

working now with the reordering of ops.  Now the problem is it adds _Copy for moves at the same level, and it also reverses the order for multiple items copied.

can't reproduce order problem, and copy fixed -- still renames unique names but user can tweak that if they want..


Navigation links are not working

Navigations links to "API properties", "distributed API Manager deployment", and "additional property" in the following page are not working.
Location : https://apim.docs.wso2.com/en/next/develop/extending-api-manager/extending-gateway/customizing-api-template-for-gateway/

4.4.0 : https://github.com/wso2/docs-apim/pull/8541
master : https://github.com/wso2/docs-apim/pull/8548


feat: init support for e2e libs with Playwright

Abstract Middleware logic from dependency on Cypress
Add support for Playwright

Fixes #116

@grantspeelman @alexeyr-ci I think the PR is ready.
It passes the CI for two envs but not the third for a weird reason.
Will investigate further tomorrow.

Amazing, nice work @KhaledEmaraDev !


CLPlatform self-assignment

Hello,
Noticed this issue by reviewing the code.
That's surely a bug:
https://github.com/nnstreamer/nntrainer/blob/89e8b7c6d5ae70d30a40cfb2d629b35138958ab4/nntrainer/opencl/opencl_context_manager.cpp#L156
platform_ = platform_;
Results in not initializing platform_ member field with platforms[0] as I suppose was intended.
Currently clang build doesn't work and I am planning to enable it for the project (at-very least for x86-64).
The reason is that clang has '-Wself-assignment' option and will be able catch typos like those.
I have not learned how to run test suit for OpenCL yet. Does the CI run any of the OpenCL tests to degree I can put my faith into?
Cheers
/P

:octocat: cibot: Thank you for posting issue #2763. The person in charge will reply soon.

Hello @piotrrak
in our github CI, you can see the build option at .github/workflows/ubuntu_clean_meson_build.yml
        meson \
          --buildtype=plain \
          --prefix=/usr \
          --sysconfdir=/etc \
          --libdir=lib/x86_64-linux-gnu \
          --bindir=lib/nntrainer/bin \
          --includedir=include \
          -Dinstall-app=true \
          -Dreduce-tolerance=false \
          -Denable-debug=true \
          -Dml-api-support=enabled \
          -Denable-nnstreamer-tensor-filter=enabled \
          -Denable-nnstreamer-tensor-trainer=enabled \
          -Denable-nnstreamer-backbone=true \
          -Dcapi-ml-common-actual=capi-ml-common \
          -Dcapi-ml-inference-actual=capi-ml-inference \
          -Denable-capi=enabled \
          ${{ matrix.meson_options }} \

It seems that the 'enable-opencl' build option is missing from the currently running CI, indicating that testing may not be taking place in GitAction.
If you have tested it locally without any issues, I will update the CI.


RDP clients render black when freerdp-shadow server runs on Windows 7.

FreeRDP server isn't working like on Linux when I run it on windows. not sure if this the current state of rendering code on windows or a bug.
How to reproduce :
1.) compile and run freerdp-shadow on windows from git.
2.) connect to the server from another pc using any RDP client.
Expected:
Use should be able to control the remote desktop with a RDP client. and renders the remote desktop.
Observed:
Mouse Events and keyboard events are received on the server side.
Client doesn't render the remote desktop only a black background.
c:\rdp\bin>freerdp-shadow.exe
[00:35:36:634] [4544:4548] [ERROR][com.freerdp.server.shadow.win] - IRDPSRAPISha
ringSession::put_ColorDepth() failure: 0x80070057
[00:35:36:634] [4544:4548] [INFO][com.freerdp.server.shadow.win] - width: 0 heig
ht: 0
[00:35:36:650] [4544:4548] [INFO][com.freerdp.core.listener] - Listening on :::3
389
[00:35:36:650] [4544:4548] [INFO][com.freerdp.core.listener] - Listening on 0.0.
0.0:3389

Other:
tested with Microsoft visual studio express 2010 (SP1), and 2012

Outdated, create a new issue if still relevant.


containers state is flawed wrt storage

Running tests with both k8s and ocic, the result is storage state is not consistent across restarts and 4/5 that results in an ocid panic. I don't really have a way to reproduce, just playing around. I tried to fix most of the panic here https://github.com/kubernetes-incubator/cri-o/compare/master...runcom:alpha (my commits on top of @nalind's #189). There are still many panics because that s.Update() at the beginning of each request makes stuff harder also, for instance, removing a container results in bad state because s.Update() does play with s.state.containers but we're also removing a container from there and that ends up in a panic..
Opened just for tracking, I don't fully have a fix for this except the commits in https://github.com/kubernetes-incubator/cri-o/compare/master...runcom:alpha (they doesn't fully fix this though)

@runcom I think we should refactor and add a separate storage interface through which all persistence calls go. It will be easier to support different backends that way as well as interoperate with kpod.

#505


connect() doesn't return EISCONN when linked to socket creation HUGE BUG

I noticed a really odd issue when writing tests for my io_uring runtime.
Namely, I'm using direct descriptors and I'm linking socket creation to a connect() I/O operation. The issue is, if I chain a normal non-linked connect() afterwards, the first run doesn't return -EISCONN. It actually returns 0! Then subsequent connect() ops will properly return -EISCONN.
I'm curious if there's just an outright bug in my code.
Here's a minimal repro with the console output:
created acceptor socket
initialized the ring
{user_data: 2, res: 0}
{user_data: 3, res: 0}
{user_data: 1, res: 0}
{user_data: 3, res: 0} # this is the offending line, should be -106
{user_data: 3, res: -106}

uname output:
exbigboss@exbigboss-ubuntu:~/cpp/connect_bug$ uname -a
Linux exbigboss-ubuntu 6.5.0-10-generic #10-Ubuntu SMP PREEMPT_DYNAMIC Fri Oct 13 13:49:38 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Build script:
clang++-17 \
  -std=c++20 -I ~/cpp/__install__/include \
  main.cpp ~/cpp/__install__/lib/liburing.a \
  -fsanitize=address,undefined \
  -o bug-test

./bug-test

main.cpp:
#include <cassert>
#include <iostream>

#include <liburing.h>
#include <netinet/in.h>
#include <sys/socket.h>

int main() {
  int ret = -1;
  auto acceptor_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  assert(acceptor_fd >= 0);

  int const enable = 1;
  ret = setsockopt(acceptor_fd, SOL_SOCKET, SO_REUSEADDR, &enable,
                   sizeof(enable));

  if (ret == -1) {
    std::cerr << "failed setsockopt() with: " << errno << std::endl;
    return 1;
  }

  sockaddr_in server_addr = {};
  server_addr.sin_addr.s_addr = htonl(0x7f000001);
  server_addr.sin_family = AF_INET;
  server_addr.sin_port = htons(42069);

  ret = bind(acceptor_fd, reinterpret_cast<sockaddr const *>(&server_addr),
             sizeof(sockaddr_in));
  if (ret == -1) {
    std::cerr << "failed to bind the socket with: " << errno << std::endl;
    return 1;
  }

  ret = listen(acceptor_fd, 256);
  if (ret == -1) {
    std::cerr << "failed to set listen() state with: " << errno << std::endl;
    return 1;
  }

  std::cout << "created acceptor socket" << std::endl;

  io_uring ioring = {};
  auto ring = &ioring;
  ret = io_uring_queue_init(128, ring, 0);
  if (ret != 0) {
    std::cout << "failed to initialize the ring with: " << -ret << std::endl;
    return 1;
  }

  ret = io_uring_register_files_sparse(ring, 1024);
  if (ret < 0) {
    std::cout << "failed to register the sparse file table with: " << -ret
              << std::endl;
    return 1;
  }

  std::cout << "initialized the ring" << std::endl;

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_accept_direct(sqe, acceptor_fd, nullptr, nullptr, 0, 42);
    io_uring_sqe_set_data64(sqe, 1);
  }

  io_uring_submit(ring);

  auto const file_index = 89;
  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_socket_direct(sqe, AF_INET, SOCK_STREAM, IPPROTO_TCP,
                                file_index, 0);
    io_uring_sqe_set_data64(sqe, 2);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK);
  }
  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK | IOSQE_FIXED_FILE);
  }
  {
    __kernel_timespec ts = {.tv_sec = 3, .tv_nsec = 0};
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_link_timeout(sqe, &ts, 0);
    io_uring_sqe_set_data64(sqe, 4);
    io_uring_sqe_set_flags(sqe, IOSQE_CQE_SKIP_SUCCESS);
  }

  io_uring_submit_and_wait(ring, 3);

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK | IOSQE_FIXED_FILE);
  }
  {
    __kernel_timespec ts = {.tv_sec = 3, .tv_nsec = 0};
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_link_timeout(sqe, &ts, 0);
    io_uring_sqe_set_data64(sqe, 6);
    io_uring_sqe_set_flags(sqe, IOSQE_CQE_SKIP_SUCCESS);
  }

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK | IOSQE_FIXED_FILE);
  }
  {
    __kernel_timespec ts = {.tv_sec = 3, .tv_nsec = 0};
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_link_timeout(sqe, &ts, 0);
    io_uring_sqe_set_data64(sqe, 6);
    io_uring_sqe_set_flags(sqe, IOSQE_CQE_SKIP_SUCCESS);
  }

  io_uring_submit_and_wait(ring, 1);

  io_uring_cqe *cqes[5] = {};
  io_uring_peek_batch_cqe(ring, cqes, std::size(cqes));

  for (int i = 0; i < std::size(cqes); ++i) {
    std::cout << "{user_data: " << cqes[i]->user_data
              << ", res: " << cqes[i]->res << "}" << std::endl;
    ;
  }

  io_uring_queue_exit(ring);
  close(acceptor_fd);
}

I've done some more experimenting.
This issue seems to replicate even when not using direct descriptors and with non-linked requests.
It doesn't necessarily seem like io_uring is emplacing the CQE too soon because accept() finishes when it should but it seems like there's a strange lag in state that's independent of timing conditions too.

@axboe let me know what you think about this! I've removed the linked timeouts and other such things as the bug still replicates.

I was actually just taking a look at it this morning. io_uring just calls the lower level connect, and for the 2nd iteration on trying to connect, the socket is in a CONNECTING state which is why it returns 0. On the 3rd attempt, it's in CONNECTED state and you get -EISCONN as expected. So not quite sure what is going on here, didn't get much further just yet but I'm pondering if a normal connect(2) would yield the same.

lmao good timing!
I actually just tried using a blocking connect(2) call and it gave me the expected results.
Output:
created acceptor socket
initialized the ring
106
106

code:
#include <cassert>
#include <iostream>

#include <liburing.h>
#include <netinet/in.h>
#include <sys/socket.h>

int main() {
  int ret = -1;
  auto acceptor_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  assert(acceptor_fd >= 0);

  int const enable = 1;
  ret = setsockopt(acceptor_fd, SOL_SOCKET, SO_REUSEADDR, &enable,
                   sizeof(enable));

  if (ret == -1) {
    std::cerr << "failed setsockopt() with: " << errno << std::endl;
    return 1;
  }

  sockaddr_in server_addr = {};
  server_addr.sin_addr.s_addr = htonl(0x7f000001);
  server_addr.sin_family = AF_INET;
  server_addr.sin_port = htons(42069);

  ret = bind(acceptor_fd, reinterpret_cast<sockaddr const *>(&server_addr),
             sizeof(sockaddr_in));
  if (ret == -1) {
    std::cerr << "failed to bind the socket with: " << errno << std::endl;
    return 1;
  }

  ret = listen(acceptor_fd, 256);
  if (ret == -1) {
    std::cerr << "failed to set listen() state with: " << errno << std::endl;
    return 1;
  }

  std::cout << "created acceptor socket" << std::endl;

  io_uring ioring = {};
  auto ring = &ioring;
  ret = io_uring_queue_init(128, ring, 0);
  if (ret != 0) {
    std::cout << "failed to initialize the ring with: " << -ret << std::endl;
    return 1;
  }

  ret = io_uring_register_files_sparse(ring, 1024);
  if (ret < 0) {
    std::cout << "failed to register the sparse file table with: " << -ret
              << std::endl;
    return 1;
  }

  std::cout << "initialized the ring" << std::endl;

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_accept_direct(sqe, acceptor_fd, nullptr, nullptr, 0, 42);
    io_uring_sqe_set_data64(sqe, 1);
  }

  io_uring_submit(ring);

  auto sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  ret = connect(sock, reinterpret_cast<sockaddr const *>(&server_addr),
                sizeof(server_addr));
  assert(ret >= 0);

  ret = connect(sock, reinterpret_cast<sockaddr const *>(&server_addr),
                sizeof(server_addr));
  if (ret < 0) {
    std::cerr << errno << std::endl;
  }

  ret = connect(sock, reinterpret_cast<sockaddr const *>(&server_addr),
                sizeof(server_addr));
  if (ret < 0) {
    std::cerr << errno << std::endl;
  }

  io_uring_queue_exit(ring);
  close(acceptor_fd);
}

Excellent! Then I don't have to test that. I'll dive into this and see what I find.

The difference is that your connect(2) test is using blocking sockets, so it waits for it. If you instantiate that socket() with SOCK_NONBLOCK, then you'd get:
created acceptor socket
initialized the ring
connect ret 115
connect ret 0
connect ret 106

which is basically what you see with io_uring - first one returns -EINPROGRESS and then get the proper connect, and then the third one will return -EISCONN. Hmm...

You could try with this:
diff --git a/io_uring/net.c b/io_uring/net.c
index 7a8e298af81b..b45ac315ef90 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -1466,8 +1466,15 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags)

 		ret = -ENOTSOCK;
 		socket = sock_from_file(req->file);
-		if (socket)
+		if (socket) {
 			ret = sock_error(socket->sk);
+			if (!ret) {
+				/* no error, mark us connected */
+				lock_sock(socket->sk);
+				socket->state = SS_CONNECTED;
+				release_sock(socket->sk);
+			}
+		}
 		goto out;
 	}

dumb question, how should I apply this patch and rebuild io_uring?

Let's hang on a bit... I'm not really convinced this is a real issue. As mentioned, if you use a nonblocking socket and do:

connect(sock, ...); -> errno == -EINPROGRESS
poll sock for POLLOUT, returns when connected
connect(sock, ...) -> ret == 0
connect(sock, ...) -> errno == -EISCONN

is the same sequence you'd see with io_uring. IOW, it behaves like a non-blocking socket would. You could of course argue that step 3 should return -EISCONN as that would be expected, but the fact is that the networking stack doesn't make that guarantee.
Not sure this is worth doing much about, or at least if we do, we should take this up with the networking side. This has probably been the behavior for decades, maybe it's even documented or discussed somewhere already.

I think this is the saner fix:
diff --git a/io_uring/net.c b/io_uring/net.c
index eb1f51ddcb23..22e6a135e5e7 100644
--- a/io_uring/net.c
+++ b/io_uring/net.c
@@ -1456,16 +1456,6 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags)
 	int ret;
 	bool force_nonblock = issue_flags & IO_URING_F_NONBLOCK;

-	if (connect->in_progress) {
-		struct socket *socket;
-
-		ret = -ENOTSOCK;
-		socket = sock_from_file(req->file);
-		if (socket)
-			ret = sock_error(socket->sk);
-		goto out;
-	}
-
 	if (req_has_async_data(req)) {
 		io = req->async_data;
 	} else {
@@ -1485,9 +1475,7 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags)
 	    && force_nonblock) {
 		if (ret == -EINPROGRESS) {
 			connect->in_progress = true;
-			return -EAGAIN;
-		}
-		if (ret == -ECONNABORTED) {
+		} else if (ret == -ECONNABORTED) {
 			if (connect->seen_econnaborted)
 				goto out;
 			connect->seen_econnaborted = true;
@@ -1501,6 +1489,16 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags)
 		memcpy(req->async_data, &__io, sizeof(__io));
 		return -EAGAIN;
 	}
+	if (connect->in_progress) {
+		/*
+		 * At least bluetooth will return -EBADFD on a re-connect
+		 * attempt, and it's (supposedly) also valid to get -EISCONN
+		 * which means the previous result is good. For both of these,
+		 * grab the sock_error() and use that for the completion.
+		 */
+		if (ret == -EBADFD || ret == -EISCONN)
+			ret = sock_error(sock_from_file(req->file)->sk);
+	}
 	if (ret == -ERESTARTSYS)
 		ret = -EINTR;
 out:

as that should yield the correct result regardless. I'll test this now and send it out.

I see. You know, I figured a lot of this might also be because I'm misunderstanding TCP/networking.
I will say this, I definitely think this defies user expectation. It's as you note, this is how normal non-blocking sockets behave but I'm not using a non-blocking socket.
Once the CQE comes in, imo, all further connect() calls should result in a EISCONN.
That being said, I have a "workaround" that gives me the "correct" output:
created acceptor socket
initialized the ring
{user_data: 2, res: 0}
{user_data: 3, res: 0}
{user_data: 1, res: 0}
{user_data: 3, res: -106}
{user_data: 3, res: -106}

Basically, I link a connect() to a connect() lmao. It ain't pretty or clean but it does the job without incurring a huge perf penalty.
using:
#include <cassert>
#include <iostream>

#include <liburing.h>
#include <netinet/in.h>
#include <sys/socket.h>

int main() {
  int ret = -1;
  auto acceptor_fd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
  assert(acceptor_fd >= 0);

  int const enable = 1;
  ret = setsockopt(acceptor_fd, SOL_SOCKET, SO_REUSEADDR, &enable,
                   sizeof(enable));

  if (ret == -1) {
    std::cerr << "failed setsockopt() with: " << errno << std::endl;
    return 1;
  }

  sockaddr_in server_addr = {};
  server_addr.sin_addr.s_addr = htonl(0x7f000001);
  server_addr.sin_family = AF_INET;
  server_addr.sin_port = htons(42069);

  ret = bind(acceptor_fd, reinterpret_cast<sockaddr const *>(&server_addr),
             sizeof(sockaddr_in));
  if (ret == -1) {
    std::cerr << "failed to bind the socket with: " << errno << std::endl;
    return 1;
  }

  ret = listen(acceptor_fd, 256);
  if (ret == -1) {
    std::cerr << "failed to set listen() state with: " << errno << std::endl;
    return 1;
  }

  std::cout << "created acceptor socket" << std::endl;

  io_uring ioring = {};
  auto ring = &ioring;
  ret = io_uring_queue_init(128, ring, 0);
  if (ret != 0) {
    std::cout << "failed to initialize the ring with: " << -ret << std::endl;
    return 1;
  }

  ret = io_uring_register_files_sparse(ring, 1024);
  if (ret < 0) {
    std::cout << "failed to register the sparse file table with: " << -ret
              << std::endl;
    return 1;
  }

  std::cout << "initialized the ring" << std::endl;

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_accept_direct(sqe, acceptor_fd, nullptr, nullptr, 0, 42);
    io_uring_sqe_set_data64(sqe, 1);
  }

  io_uring_submit(ring);

  auto const file_index = 89;
  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_socket_direct(sqe, AF_INET, SOCK_STREAM, IPPROTO_TCP,
                                file_index, 0);
    io_uring_sqe_set_data64(sqe, 2);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK);
  }
  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK | IOSQE_FIXED_FILE);
  }
  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_IO_LINK | IOSQE_FIXED_FILE |
                                    IOSQE_CQE_SKIP_SUCCESS);
  }
  {
    __kernel_timespec ts = {.tv_sec = 3, .tv_nsec = 0};
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_link_timeout(sqe, &ts, 0);
    io_uring_sqe_set_data64(sqe, 4);
    io_uring_sqe_set_flags(sqe, IOSQE_CQE_SKIP_SUCCESS);
  }

  io_uring_submit_and_wait(ring, 3);

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_FIXED_FILE);
  }

  {
    auto sqe = io_uring_get_sqe(ring);
    io_uring_prep_connect(sqe, file_index,
                          reinterpret_cast<sockaddr const *>(&server_addr),
                          sizeof(sockaddr_in));
    io_uring_sqe_set_data64(sqe, 3);
    io_uring_sqe_set_flags(sqe, IOSQE_FIXED_FILE);
  }

  io_uring_submit_and_wait(ring, 1);

  io_uring_cqe *cqes[5] = {};
  io_uring_peek_batch_cqe(ring, cqes, std::size(cqes));

  for (int i = 0; i < std::size(cqes); ++i) {
    std::cout << "{user_data: " << cqes[i]->user_data
              << ", res: " << cqes[i]->res << "}" << std::endl;
    ;
  }

  io_uring_queue_exit(ring);
  close(acceptor_fd);
}

I posted a fix here:
<EMAIL_ADDRESS>which should do the right thing for both the original issue that actually caused this change, and handle your case as well. Before that referenced bluetooth issue, io_uring did do the double connect itself. With the above, we do that again, but just ensure we handle potential issues that can arise from that. Eg your above comment will fail on bluetooth. Probably not an issue for you, but it is for me as it needs to work everywhere.

I'll get this fix into 6.7-rc1 and ensure it rolls back to stable as well.

Alright, that sounds perfect.
I really appreciate you taking the time to look into this.
I'll go ahead and close the issue. Users have a workaround for now and it sounds like the issue will eventually resolve itself. There's nothing else to be done. Thank you for your time.


No partition table found.

I am trying to inspect my image, but am receiving a PartitionError.
% ./mp-image-tool-esp32 ../micropython/ports/esp32/build/micropython.bin
Opening image file: ../micropython/ports/esp32/build/micropython.bin...
Chip type: esp32
Flash size: 16MB
PartitionError: No partition table found.
Traceback (most recent call last):
  File "mp-image-tool-esp32/./mp-image-tool-esp32", line 15, in <module>
    sys.exit(main())
             ^^^^^^
  File "mp-image-tool-esp32/./mp-image-tool-esp32", line 11, in main
    return src.mp_image_tool_esp32.main.main()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "mp-image-tool-esp32/src/mp_image_tool_esp32/main.py", line 285, in main
    raise err
  File "mp-image-tool-esp32/src/mp_image_tool_esp32/main.py", line 279, in main
    process_arguments()
  File "mp-image-tool-esp32/src/mp_image_tool_esp32/main.py", line 150, in process_arguments
    table: PartitionTable = copy.copy(image.table)
                                      ^^^^^^^^^^^
  File<EMAIL_ADDRESS>line 1001, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "mp-image-tool-esp32/src/mp_image_tool_esp32/image_file.py", line 200, in table
    table.from_bytes(data)
  File "mp-image-tool-esp32/src/mp_image_tool_esp32/partition_table.py", line 169, in from_bytes
    raise PartitionError("No partition table found.", self)
src.mp_image_tool_esp32.partition_table.PartitionError: No partition table found.

My typical build artifacts include:

bootloader.bin
micropython.bin
ota_data_inital.bin
partition-table.bin

Partition table is:
#   Name,      Type,     SubType,    Offset,         Size,     Flags
     nvs,      data,         nvs,          ,     0x004000,
 otadata,      data,         ota,          ,     0x002000,
phy_init,      data,         phy,          ,     0x001000,
   ota_0,       app,       ota_0,          ,     0x270000,
   ota_1,       app,       ota_1,          ,     0x270000,
     vfs,      data,         fat,          ,     0xb00000,

The micropython.bin file in the build directories are what espressif calls an "app image" file. They include only the micropython firmware and do not include the bootloader or partition table.
The firmware.bin files combine the bootloader, partition table and micropython.bin file into a full firnware foir writing to a new device.
Try ./mp-image-tool-esp32 ../micropython/ports/esp32/build/firmware.bin instead.

Makes sense.  Interesting, my build does not produce a firmware.bin.
idf.py -D MICROPY_BOARD=$BOARD_NAME -D MICROPY_BOARD_VARIANT=$VARIANT_NAME -B $BUILD_FOLDER_NAME build

Ok, I used make instead of idf.py, which did produce a firmware.bin but same No partition table found error.

Ooh. That's odd. You should use the make command rather than idf.py to build micropython. You may need to rebuild from scratch (eg. "make clean"). See the instructions in ports/esp32/README.md.

I did, but let me try flashing the built firmware.bin to make sure it actually works.

I am closing this issue as I have been unable to replicate and there is nho new information on the bug.
If you are still getting "No partition table found." errors, please re-open and provide additional information.


Comment: freeze ClusterAutoscaler configuration by MachinePool contro…

…ller
The MachinePool controller should not be in the business of configuring the (singleton) ClusterAutoscaler. We've got some minimal setup in the code that's considered necessary if the (any) MachinePool has Autoscaling enabled; but further configuration should be left to the user -- whether in-cluster or from the hub via syncsets.
HIVE-2220

/hold pending outcome of discussions in HIVE-2220

We're going a different direction via #2025 or similar. See HIVE-2220 for details.


[0.10] Add protocol and auth trait requirements

Protocol traits now list the traits that protocol implementations must
understand in order to successfully implement the protocol. This
provides the ability to better ensure correctness of protocol
implementations as protocols evolve.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Should authDefinition receive the same treatment, allowing aws.auth#sigv4 to specify it utilizing aws.auth#unsignedPayload?

Sure. I updatred the auth trait to also have a list of trait requirements. We discussed this and don't see it as a general abstraction at this point (like a new property of trait definitions). These requirements are really a function of serialization and deserialization specific traits, which is what auth and protocol traits are.


path=. confusion

Just realized that this could have a potential conflict with the documented usage to use path: ., since a .git directory will likely already exist at ./.

Originally posted by @spenserblack in https://github.com/spenserblack/actions-wiki/pull/70#discussion_r1186722661

we could use:

GIT_DIR is the location of the .git folder. If this isn’t specified, Git walks up the directory tree until it gets to ~ or /, looking for a .git directory at every step.
GIT_WORK_TREE is the location of the root of the working directory for a non-bare repository. If --git-dir or GIT_DIR is specified but none of --work-tree, GIT_WORK_TREE or core.worktree is specified, the current working directory is regarded as the top level of your working tree.
from https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables

and basically do
export GIT_DIR=../wiki.git
export GIT_WORK_TREE="$INPUT_PATH"
gh repo clone "$GITHUB_REPOSITORY" "$GIT_DIR" -- --bare
git config --unset core.bare
git reset

?? 🤔

There's a working version of a github wiki action that uses this technique: https://github.com/Andrew-Chen-Wang/github-wiki-action/tree/jcbhmr
ref https://github.com/Andrew-Chen-Wang/github-wiki-action/issues/30

I'm thinking the solution to this is really simple, actually. When path is . (or something that resolve to .), just push the main repository's history to the wiki. The whole point of pulling down the wiki's commits is to preserve its history. When the main repository is the wiki, that's not really necessary.
This can be even further simplified, but I'll put that in a separate issue.

Good point! Would this work with an ignore: input though? 🤔 #49

Since nobody has actually expressed a need for #49, I think it's not that high priority. As a potential bug, this takes priority IMO.
But I believe we can find an implementation of #49 that wouldn't conflict with this :slightly_smiling_face:


chromium: 124.0.6367.91 -> 124.0.6367.118, lower version range of --ozone-platform-hint patch

Description of changes
https://chromereleases.googleblog.com/2024/04/stable-channel-update-for-desktop_30.html
This update includes 2 security fixes.
CVEs:
CVE-2024-4331 CVE-2024-4368
Ref #306010
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Forgot about ungoogled-chromium not being in sync with chromium

Result of nixpkgs-review pr 308358 run on x86_64-linux 1

20 packages built:

aws-azure-login
chromium
chromium.sandbox
fast-cli
google-chrome
mermaid-cli
mermaid-filter
netflix
percollate
playwright-test
puppeteer-cli
python311Packages.pytest-playwright
python311Packages.pytest-playwright.dist
python312Packages.pytest-playwright
python312Packages.pytest-playwright.dist
sharedown
single-file-cli
sitespeed-io
vhs
wayback

Did some manual testing under wayland and the usual:
# git describe --abbrev=40
24.05-pre-68219-ge32981ab32c921f0506a3a560512e3a34726eafe
# nix-build -A stable nixos/tests/chromium.nix
/nix/store/4v39lx2gkvw7ffbp6k0zbbg6k5kfb12c-vm-test-run-chromium-stable


[Feature Request] 32-bit HDR Output Workflow

I managed to use this extension to create a passable approximation of a high-dynamic range image in 32 bit per channel. I must use photoshop to do some of the steps, but most of the hard work is done with this extension.
I would like to describe my workflow and have your input but I am not sure if this "issue" tab is the right place to do it.
There are also a few feature requests I have in mind to help me with this HDR generation process, but let's describe my current workflow prototype first and see what you think.
I'm also working on panoramic HDRI environment maps, which are quite useful for real-time 3d engines. More on that soon !
Finally, THANK YOU SO MUCH for making this extension. I was already a fan, and I had already tried to do what I just did today with an earlier version of your extension, but it looks like the latest version has the features that were missing to make this whole HDR image generation work. You made this possible by creating and sharing this extension, and I can't thank you enough for that.

Discussing here is fine I guess.
And thank you for the positive comments~

I myself have thought about generating actual HDR image as well.
But afaik, there is no Python package that can export native HDR images with 32-bit per channels, at least I wasn't able to find any.
So 3rd-party tool, like Photoshop, is still needed I think...

I will document my little process here in this thread then. I'll go prepare that right now, I'll reuse the prototype I worked on yesterday as an example, so it should not be too long.

How to use the Vectorscope extension for A1111 to create 32 bit HDR images
Create an image as you normally would. This will be our reference image in 8 bit. Here is the picture I used, and, just underneath, the prompt I used.

Cute creature from Space. terraforming. Alien Flora, Miki Asai Macro photography, close-up, hyper detailed, trending on artstation, sharp focus, studio photo, intricate details, highly detailed, by greg rutkowski
detailed face, detailed skin
<lora:xl_more_art-full_v1:1>
Negative prompt: ugly, tiling, poorly drawn hands, poorly drawn feet, poorly drawn face, out of frame, extra limbs, body out of frame, blurry, bad anatomy, blurred, watermark, grainy, signature, cut off, draft,  text, logo
Steps: 30, Sampler: DPM++ 2M Karras, CFG scale: 7, Seed:<PHONE_NUMBER>, Size: 768x1344, Model hash: 730028c80f, Model: sdxlYamersPhotorealistic_version1, RNG: CPU, CFG Rescale phi: 0, Lora hashes: "xl_more_art-full_v1: fe3b4816be83", SGM noise multiplier: True, Version: v1.6.0-127-g102b6617

(directly inspired from this post on civitai: https://civitai.com/images/2287991?modelVersionId=152309 big thanks to ledadu)
Once you have this picture that you like, you must set the seed to be the same as the one used to generate the picture. You can use the little "recycle" icon next to the seed # parameter to do that automatically with the currently selected picture.
Now we will create variations of that picture, but with different exposures. This technique, called bracketing, was used to create actual HDR content without an HDR capable camera. Nowadays, you have an HDR camera in your phone !
To do that we will activate the Vectorscope extension and set it like this:

Basically, in this Vectorscope panel, you want to check enable and alt and set the brightness parameter to a value of minus six -6.0
Now you can press the big "Generate" button to generate our first bracket, the darkest of them all.
In my case, this gives me something that looks like this:

Once this is done, you can go back to the Vectorscope panel and set the brightness to minus four -4.0 . And press generate again. This provides our second bracket, which is slightly brighter than the previous one, but still dark.
And then we repeat those steps of adding 2 to the previous brightness value and pressing generate 5 more times, so we get brightness=-2.0, brightness=0.0, brightness=2.0, brightness=4.0, and, finally, brightness=6.0.
You should now have 7 images total, from the darkest (brightness=-6.0) to the brightest (brightness=+6.0), with your original image in the middle of the series (brightness=0.0). For reference here is the brightest I got for the current example:

Now we move to photoshop. We don't open or create a new document, but instead go into file-automate-merge to HDR pro...
Once this is started, you will get access to a little interface where you can load the 7 pictures we just generated - our 7 brackets. After browsing to the proper folder and selecting your 7 pictures, you will get something that looks like this:

Press OK, and you will get to the next page, where we will be assigning exposure values to each of our 7 pictures. You could set the EV parameter directly, but I like to use ISO values instead. Here is a reference chart for these:
brightness=-6.0  /  ISO =   25
brightness=-4.0  /  ISO =   50
brightness=-2.0  /  ISO = 100
brightness= 0.0  /  ISO =  200
brightness= 2.0  /  ISO =  400
brightness= 4.0  /  ISO =  800
brightness= 6.0  /  ISO = 1600

The interface will look like this:

Once all the ISO values have been attributed, you can press OK and get to the next step, which brings you to a new screen, like this:

First, make sure you are in 32 bit mode. Also, make sure the option to is unselected. After that you get to set the white point. You basically have to move the little triangle slider under the histogram on the upper right so that the brightest part of your image show up as white, but just barely. You should get something like this:
What you really don't want to have is something like this, with overblown white values:

Once the white level is set, you can press OK and you will get your 32 bit per color channel HDR picture.

Yes, it is very dark, and this is normal. To make this apparently too dark HDR picture usabel we will apply a gamma correction to bring that gamma value from linear (1.0) to exponential with a new value of 2.2. And then, finally, we will add some exposure.
Here is the result with the 2,2 gamma correction

And here it is with boosted exposure - here a value of 5. By the way you can also play with the offset value to change the contrast.

That's it, you now have a usable HDR version of your initial image ! Test the new dynamic range available to you by adding a level, curve or exposure effect, if you want to stay in photoshop. Or bring it into a 3d app, apply it as a texture map, and see how much richer it is when it reacts to lighting compared to an 8 bit version. Or send it to an HDR display to see it in its full HDR glory, which normally looks closer to a photographic diapositive, or an analog film frame, than to a video frame.

Next tutorial: how to use Vectorscope to generate panoramic HDRI environments.
And in case you are wondering, I tested the above workflow with the highres fix and it works marvelously well, and it's also fully compatible with Tiled VAE, both of which are essential features for generating high resolution panoramas.

Currently writing a Script to do this workflow
Should be prossible to do this in 1 Generation (So you don't need to press Generate 7 times)

Please check it out~

I just got my first TIFF in 16 bit per color channel using your HDR Script and it seems to work really well.
I will make more tests to make sure everything works as it should, and I will report back if I find anything.
Thanks again !


Results Grid Is Slow and Clunky to Scroll When Results Contain XML

Happens on both my macbook in macOS and Windows machine.
I think this problem rears it's head when the grid has XML in it. Although it seems to truncate it for display it still seems to slow scrolling to a crawl.
For example try running something like sp_blitzwho (https://github.com/BrentOzarULTD/SQL-Server-First-Responder-Kit/blob/dev/sp_BlitzWho.sql) which returns XML and large query fields.

@gavdraper For this type of use case, can you suggest your expectation by assuming that rendering large data has a perf impact. In this case, instead of render out the actual XML data, would it be better to display <xml_data> as a link and click to open the actual data in an editor and increase the grid rendering speed? This type of issue can occur for a large binary data as well. We are looking for a generic UX as well as improving the rendering performance itself.

A link to the actual data and a fast performing result grid would be great. These large XML fields for me at least are far more useful as links to the data than trying to see it all in a grid.

Probable similar root cause as issue https://github.com/Microsoft/sqlopsstudio/issues/1119.