BAU: fix Bolton Council name to match spreadsheet

Change description
fix Bolton Council name to match spreadsheet. See discussion on Slack for further context. This is safe because no data has been submitted under the previously defined name.

[ ] Unit tests and other appropriate tests added or updated
[ ] README and other documentation has been updated / added (if needed)
[x] Commit messages are meaningful and follow good commit message guidelines (e.g. "FS-XXXX: Add margin to nav items preventing overlapping of logo")

How to test
User should be allowed to upload Bolton spreadsheet as currently defined.
Screenshots of UI changes (if applicable)

pre-commit.ci autofix


New Youtube refs/heads/youtube-tanzu-talk-343902679 Episodes

Automatically Generated by GitHub Action

Subsumed by https://github.com/vmware-tanzu/tanzu-dev-portal/pull/384


LTE Ecosystem

why we have a admin lte ecosystem, like example that work very nicely, that more than current default, contribute by anyone
like better form data entry for purchase order, invoice, shipping, or any thin
work with balsamiq may be will bring more better ecosystem, or another mockup apps

Hello @fthamura,
AdminLTE is a simple HTML template. Unfortunately, we don't have the time to create a complete solution or integrate it with anything on that scale. Sorry for the inconvenience.
Thanks!


add more tests

I think it is useful. If @seanmonstar and @reem want to use it I will write some end-to-end tests using it.

@winding-lines this would be useful. You could look at the coveralls report to see where some test cases are missing...

@seanmonstar sounds good. I have been looking a bit at adding proxy to client but it would be useful to know more of the code first :+1:


fix: typo

Description
Fixed a little typo in the README.md
Checklist

[ ] Added link to Github issue or Trello card
[ ] Added tests

Codecov Report

Merging #218 (2dbace6) into master (23d17d3) will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #218      +/-   ##
==========================================
- Coverage   84.74%   84.71%   -0.03%
==========================================
  Files          40       40
  Lines        2052     2048       -4
==========================================
- Hits         1739     1735       -4
  Misses        313      313

Impacted Files
Coverage Δ

wellcomeml/ml/frequency_vectorizer.py
79.24% <0.00%> (-0.76%)
:arrow_down:

wellcomeml/ml/bert_vectorizer.py
96.00% <0.00%> (-0.16%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 23d17d3...2dbace6. Read the comment docs.

Oh hello @ivyleavedtoadflax. Thanks for catching that extra space.


Can't build project on Xcode 12 beta

New Issue Checklist

[x] I have read and understood the CONTRIBUTING guide
[x] I have read the Documentation or CocoaDocs
[x] I have searched for a similar issue in the project and found none

Issue Info

Info
Value

Platform Name
iOS

Platform Version
14.0

CocoaLumberjack Version
3.6.2

Integration Method
CocoaPods

Xcode Version
Xcode 12 beta 4

Repro rate
100%

Issue Description and Steps
Xcode 12 beta 4 can't find the module CocoaLumberjack. Please add support of it.

For Xcode 12 support, you need to use the master branch.
Also, there is #1161 which might be the same issue.

I am also facing the same issue. Please check
Showing All Messages
~/Logging Manager/LogManager.h:15:9: 'CocoaLumberjack/CocoaLumberjack.h' file not found

Failed to emit precompiled header '~/Xcode/DerivedData/ABC-fgropwrduxzlubdcsarmrrumkcwd/Build/Intermediates.noindex/PrecompiledHeaders/App Stage-Bridging-Header-swift_26K6PXALQ98NL-clang_1EJ4IRL0H586Y.pch' for bridging header '~/Models/App Stage-Bridging-Header.h'

@shikhil-sevenlakes Can you provide a few details? What integration method are you using? What "version" of CocoaLumberjack (you should be using the current master branch).

@shikhil-sevenlakes Can you provide a few details? What integration method are you using? What "version" of CocoaLumberjack (you should be using the current master branch).

Info
Value

Platform Name
iOS

Platform Version
14.0

CocoaLumberjack Version
3.6.2

Integration Method
CocoaPods

Xcode Version
Xcode 12 beta 5

I am using the master branch -
pod 'CocoaLumberjack/Swift',:git => 'https://github.com/CocoaLumberjack/CocoaLumberjack.git', :branch => 'master'

@ffried - I am still facing the same issue in Xcode 12 beta 6 as well.

Now I confirmed I was able to build a small sample project below.
So I don't think this is directly related to CocoaLumberjack's issue.
Please confirm you set SWIFT_VERSION to your Xcode project. Or can you make a small project for repro?

Info
Value

Platform Name
iOS

Platform Version
14.0

CocoaLumberjack Version
master branch

Integration Method
CocoaPods 1.9.3

Xcode Version
Xcode 12 beta 6

Podfile
platform :ios, '10.0'

target 'SampleSwiftApp' do
  use_frameworks!
  pod 'CocoaLumberjack/Swift', :git => 'https://github.com/CocoaLumberjack/CocoaLumberjack.git'
end

ViewController.swift
import UIKit
import CocoaLumberjack

class ViewController: UIViewController {
    override func viewDidLoad() {
        super.viewDidLoad()
        dynamicLogLevel = .info
        DDLog.add(DDOSLogger.sharedInstance)
        DDLogVerbose("verbose")
        DDLogDebug("debug")
        DDLogInfo("info")
        DDLogWarn("warn")
        DDLogError("error")
    }
}

@sushichop Thanks for confirming. I was just trying to do the same, but failed at installing CocoaPods (which seems to be related to Big Sur, though).

@ffried I see! My environment is Catalina(mac OS 10.15.6)🙂

@ffried @sushichop  - Thanks. The SWIFT_VERSION of the project is Swift 5.
I am able to build the app for the device but it fails to compile for simulators. It gives header file not found.
Not sure if this is related to Cocoalumberjack or I need to change some project settings.

I am having the same problem too but with SPM.

@glorysaber - I was able to resolve this by this method - https://stackoverflow.com/questions/63452257/xcode-12-beta-4-lexical-or-preprocessor-issue-header-file-not-found-simulato/63639145#63639145

@shikhil-sevenlakes thanks for the link, but in this case it was not enough. Lumber Jack seems to be the only framework I am still having an issue with.

@shikhil-sevenlakes Thanks! It actually solved my issue, turns out I had missed on of my dependent targets.

I'm closing this since as far as I've seen the issues collected here have either been resolved, or aren't on our side.
If anyone still has issues, please open a new issue.


python3Packages.tensorflow: move asserts to broken to unbreak eval

Description of changes

Asserts can break eval and cause nixpkgs-review to fail to run. This PR moves the asserts into broken to avoid breaking eval.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I didn't check but I think that broken would be evaluated before cudnn so we should be OK

Definitely; all the values are evaluated lazily and meta.broken should be evaluated way before anything where cudnn is actually used.


Extract audio and keep the video file, but nothing else

Checklist

[X] I'm reporting a feature request
[X] I've looked through the README
[X] I've verified that I'm running yt-dlp version 2022.02.04. (update instructions)
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Description
I'd like to download a video (A.mkv) and extract its audio as a separate file (A.opus).
The -x option extracts the audio and gives me a clean audio file, but the final video (A.mkv) is deleted.
So I have to use the -k option to keep the intermediate video. This gives me the final audio and audio+video versions (A.mkv and A.opus), but also gives me all of the intermediate files which include a number of audio and video files that have been used to create the previous files. Also, the cover art and subtitles are preserved.
Would it be possible to have a parameter for downloading the video and extracting the audio from it and keeping ONLY these two?

You could do -x -k --exec "del %(requested_formats.:.filepath)#q"
(On UNIX, change del to rm)

Also, the cover art and subtitles are preserved.

That is because of other options in your config, not -k

@pukkandan
Thanks!
I looked again and you are right! Unfortunately, I couldn't get the exec command to work and will be using a separate command to convert using ffmpeg.


Quake Mode multi monitor setup

I've been toying with quake mode and noticed some odd behaviors when running on multiple monitors, where one is on a vertical orientation.  On first summon quake mode dropped down from my desired "main" monitor.  Consecutive summons have it split between two monitors but never on a single monitor.  I tried changing the monitor and desktop values of the command without luck.
In the attached image you can see that the window dropped down to my left most monitor (which is vertical).
It appears the drop down is tracing location of mouse relative to left|right of center in regards to the entire resolution of the combined screens (?).
If my mouse is towards left of center it will render on left most monitor, with the right edge meeting center of main screen.

If my mouse is to the right of center of main screen it renders on the right "half" of the resolution, with the left edge meeting center of main monitor.

Last interesting thing to happen is -- if my mouse is on the left edge of my left most screen it renders off screen.

In my case, I got two monitors. On first I have no Windows taskbar, on second I got Windows taskbar located at right monitor edge.
When I run Quake mode on monitor 2 it is looking ok:

But when I do Quake mode on monitor 1, Windows Terminal is having gap in place where the Windows taskbar is on second monitor:

I've been having some issues with the quake mode on multi monitors. (love this new feature btw)..
My setup is a 4k landscape monitor above the laptop and a hd portrait monitor next to the that. All 3 monitors have different resolutions and scaling factors.
When opening on the laptop screen it works perfectly, full width of the screen.
When opening on the 4k screen, it only goes to 3/4 width of the screen.
When Opening on the 1080, portrait screen, its width spills over to the 4k screen.
Sometimes it gets confused and doesn't know which is the active screen and opens in the previous window, other times it opens in random places on the screen (usually the middle)
There is also what feels like screen tear in the slide down effect.
Hope this helps in getting the bugs ironed out. If you want I provide screenshots.
Brent

Welp, that's weird. I've been using QM for a while with multi-mon, but never ran into anything like this. Maybe the vertical/horizontal orientation is important here. I'll stick this on 2.0 for now, but if anyone who's actually hitting this could help debug, that'd definitely be appreciated.

Ok I think I can reproduce my issue now. Like I said before, I have two monitors, both have the task bar at the bottom, and both are using the regular orientation.

Open QM terminal with hotkey
Hit winkey+shift+[left or right arrow key] to move it to the other monitor
The terminal now has the wrong size in the second monitor

It is already wrong now, but if you move it back to the first screen (With either the QM terminal hotkey or winket+shift+[left or right arrow key] ), it is also wrong on the first monitor (It ends up like my screenshot in the comment above)

notes before I leave for the weekend:

WM_WINDOWPOSCHANGING is probably not the right place to do this.

It gets called way too often, we don't need to try and enterQuakeMode that many times
For whatever reason, doing enterQuakeMode there does calculate the right size, but then the window still ends up at the wrong dimensions. Maybe this is  getting followed up with a WM_SIZING or something.

Where is the right place to determine that we've moved? WM_MOVING? I suppose that might work

IM having a similar issue. I have horizontal screens layout but they're different resolutions and scaling. My secondary screen is 4k@150% scaling and my primary screen is 1440p. When the terminal opens on the secondary screen it doesn't go all the way across.

PS - Also really appreciate this quake console mode. I was doing it manually with AHK and then it wasn't working properly and realized you guys backed the feature in using the same hotkey 😅

Actually, is there any way to just turn this off? I want my own implementation back and WT won't let me use this keyboard shortcut for my AHK script anymore.
I thought it was cool but Quake mode is a hindrance I want it gone.


Mask Input: Displayed value doesn't update on form reset

Reproduction
https://formkit.link/8a9ccc71c33b3b92d946b1475a9b5736
Describe the bug
Using reset() on a mask input resets the value but doesn't reset the user-entered "masked value" in the input.
In the example.
To reproduce, open the reproduction link, type something in the two fields, and press the Reset button. The value in the mask field (e.g. #ff0000) remains when it should be removed
Environment
• OS: Mac
• Browser: any
• Version: latest

+1

I noticed  another issue related to form reset (https://github.com/formkit/formkit/issues/1509) was recently closed. Could it be related to this one?


Create documentation for adding boxsets

would you say this ultimate edition would be a boxset? has some movies included so I assume so

If it spans multiple movies then I consider it a box set. If it is just one movie but multiple discs then it is just a movie.
Is Neon Genesis Evangelion a TV series? If so it would be added as a series

Yeah, it's a tv show + 2 movies in the set (2 versions of the tv show + 3 versions of the movies) so a box set. I'll wait til the documentation for box sets is done though.

Yeah, that sounds like a good candidate for a boxset. I haven't tried mixing a series and movies into one boxset so this should be a good test case. I am pretty sure it will work. I will try getting a first version of the documentation done in the next couple of days. Thanks!

@daynyxx - here is some documentation on boxsets: https://github.com/lfoust/thediscdb-data/wiki/Importing-Boxsets. Let me know what you think or if you have any feedback (on either the documentation or the whole boxset import process). Thanks!


[BUG] Links in section titles

I have section titles that contain links to other notes or PDF documents. When reviewing cards, the plugin does not correctly display such sections titles. The correct behaviors should be to show only the text after the vertical bar |. See screenshot below.

Hi @dengrish seems like a good suggestion to me.
So you would want to see the question rendered as:
3. Classification > Performance Measure > Precision and Recall
Is that correct?
Personally I can't see anyone wanting the current formatting and I think the code should simply be modified.
If anyone thinks differently, please comment and perhaps the formatting style would be made a new option.
Cheers
Ronny

Yes! It should be rendered as 3. Classification > Performance Measure > Precision and Recall. Thanks!

1. The ML Landscape.md
A sample note is attached. Thanks!

Created a Fix for this now its up to the reviewer to include it.


Support for navigating class inheritance hierarchy

We need to :

get direct superclass
get direct subclasses
other related methods to navigate the inheritance hierarchy we have in the Pharo reflection API

Maybe this provides a building block towards a solution?
subclassesOf: baseClass
<javascript:
'var allVars = Object.keys(Smalltalk);
var classes = allVars.filter(function (key) {
try {
var obj = Smalltalk[key];
return obj.prototype instanceof baseClass;
} catch (e) {
return false;
}
});
return classes.map(clName => Smalltalk[clName]);'>


linux seccomp extension for glibc

r? @JohnTitor
(rust-highfive has picked a reviewer for you, use r? to override)

@bors r+

:pushpin: Commit 8c5c3404aef2b9aba5d3405b59da5a8cac889fe4 has been approved by Amanieu

:hourglass: Testing commit 8c5c3404aef2b9aba5d3405b59da5a8cac889fe4 with merge 665031b3845299e167c6878b718f113cab9c2165...

:broken_heart: Test failed - checks-actions

@bors r+

:pushpin: Commit 621a3c2a54bb6c03760bfb0c84e86129be83a315 has been approved by Amanieu

:hourglass: Testing commit 621a3c2a54bb6c03760bfb0c84e86129be83a315 with merge 008f1f05bbf0a5085a9ef2f10499cb2142e3a0de...

:sunny: Test successful - checks-actions, checks-cirrus-freebsd-11, checks-cirrus-freebsd-12, checks-cirrus-freebsd-13
Approved by: Amanieu
Pushing 008f1f05bbf0a5085a9ef2f10499cb2142e3a0de to master...


Is Alluxio have the docker image that can support cephfs as backend?

I want to use fluid + alluxio to accelerate the cephfs read.
Is there docker images that can use:
2022-12-20 12:46:44,948 WARN  UnderFileSystem$Factory - Failed to create UnderFileSystem by factory alluxio.underfs.cephfs.CephFSUnderFileSystemFactory@6ee7992a: java.lang.NoClassDefFoundError: com/ceph/fs/CephFileAlreadyExistsException
2022-12-20 12:46:44,948 ERROR FileSystemMasterClientServiceHandler - Exit (Error): Mount: request=alluxioPath: "/cephfs"
ufsPath: "cephfs://<IP_ADDRESS>\;<IP_ADDRESS>\;<IP_ADDRESS>/"
options {
readOnly: false
shared: false
commonOptions {
syncIntervalMs: -1
ttl: -1
ttlAction: FREE
}
}

It seem that you can give some more suggestion.
@maobaolong

Alluxio dockerfile use Alpine Linux, but I can't find package on apk package manager...
I try to use ubuntu 20.04 to build the alluxio images, but it seem can't use.
alluxio-cephfs.zip
Alluxio support cephfs:
https://docs.alluxio.io/os/user/2.8.1/cn/ufs/CephFS.html

@andyzheung are you using the Alluxio official docker image which include the Cephfs binaries?
@ssz1997 Does fluid docker image contain exactly the same UFS binaries in the lib/ directory?
Try to figure out it's a fluid alluxio docker image problem or a general alluxio docker image problem.

@LuQQiu The alluxio image used by Fluid is built by running the dev/scripts/generate-tarballs single script in Alluxio repo, so theoretically all the binaries are the same.
In reality the alluxio-underfs-cephfs-hadoop-2.8.1.jar in the Alluxio official tarball is 272 Kb larger than the tarball generated by Fluid.


test issue

feel free to close

feel free to close again

this shouldn't trigger

this should make the bot feel free to close


build: external_cmake for cmake deps.

This revives #4516, replacing all external cmake deps with the pending bazel foreign_cc support.
To make use of this, with the current release Bazel version, it's necessary to set
--experimental_cc_skylark_api_enabled_packages=@rules_foreign_cc//tools/build_defs,tools/build_defs
on the CLI or in .bazelrc.
Risk Level: Low
Testing: bazel test //test/...
Signed-off-by: Harvey Tuch<EMAIL_ADDRESS>

This PR requires changes to .bazelrc still for the experimental flags. Question; should we provide a fallback to the deprecated build recipes if missing? The downside of this is it might involve Bazel hackery and would have two sources of truth for the build in the interim.
CC @dprotaso @moderation @irengrig

@sesmith177 @mattklein123 PTAL. I think we should ship with the required additional CLI args. This will break anyone not using the repo's bazel.rc, but we can do a mail to envoy-dev warning folks. The advantages are we can move towards Windows support, unblock the mobile client work for next year and also effectively retire most uses of build recipes.

@mattklein123 gperftools is being tracked at https://github.com/gperftools/gperftools/issues/800. Doesn't look like much recent progress. I will ping the thread.
I just synced with Bazel team, who say that we will be able to use external_cmake without CLI flags in 0.22, which will happen mid-late Jan. @sesmith177 @mattklein123 thoughts on whether we just wait until then rather than going back-and-forth with the dev community on this?

Since Envoy having "real" support for Windows is blocked on https://github.com/bazelbuild/bazel/issues/5163, we're OK with waiting until January. This specific PR isn't going to block us, since we have these changes on our fork anyways

Sure, waiting till January is fine with me.

This needs to be rebased to pull BoringSSL FIPS (which, even though is built using CMake, should continue to be build that way using the provided script and not cmake_external).

@PiotrSikora for BoringSSL FIPS, I think if we're going to keep it script based in the long term, it should be moved to https://github.com/envoyproxy/envoy/blob/master/bazel/EXTERNAL_DEPS.md#adding-external-dependencies-to-envoy-genrule-repository style of external repo. We want to completely retire build recipes and the repository rules.
Will you do the migration or should we file an issue?


Close viewHTML mode

Hi
How can i close editor viewHTML mode when blur on editor?

Hey!
Try something like:
$editor.on('tbwblur', function () {
	if ($(this).parent().hasClass('trumbowyg-editor-hidden')) {
		$(this).trumbowyg('toggle');
	}
})

Hope it helps :)

Thank you! But this is not working for me, maybe because i use editor in the iframe.

Did you put this code in the iframe?


Watch not working, Needed a small change.

It worked when
var paths = {
sass: ['./scss/**/*.scss']
};
changed to
var paths = {
sass: ['./www/**/*.scss']
};
Anybody else have the same issue?

Hi, you shouldn't keep scss in your www folder. WWW folder is kind of your dist folder, you keep there your almost final product. Sources should be kept separately to output.
Default Ionic library sets watch on scss folder to keep it well organized for beginers, and to not mix source with output.
Everything more with customization you have to do in your own project repository :) It isn't library's fault.
Everyone needs a slightly different configuration to work better and faster, one has a habit to keep JS and SCSS sources in 'src' folder, another person wants to keep it in res/js & res/scss, and another - like for example you want to keep source files in www folder.
Move your .scss files into /scss folder and you're good to go.
IMO best approach is to keep every sources in other folder than www, e.g. in 'src' folder with subfolders for each language, and generate www folder dynamically using gulp, I would suggest even adding www to .gitignore in this case.

scss should not be placed in the www directory.

@michalsnik Thank you very much for your details reply. :) IMO, The example, should be working perfect when someone want try ionic for the first time. As you said, it is best to keep scss and js files in src folder., then the default gulp task should minify and pack them in the www folder.
I didn't get a good reason for keeping scss files in www, and watch tasks pointing to another folder in a freshly cloned copy.
( Is things are mentioned somewhere on docs.? I didn't see them! )


Generate age distributions per state

Use census data to generate age distributions per state, this will enable us to get integrated into unified UI quickly

Do we have census sources we want to use? Are we looking to target just the USA or other countries as well? Where does this integrate into the Sim?

Grace Huynh from Microsoft suggested this source:
https://data.census.gov/cedsci/table?g=0100000US.050000&layer=VT_2018_050_00_PY_D1&tid=ACSST5Y2018.S0101&vintage=2018&hidePreview=false&t=Age and Sex%3APopulations and People

Would we want to pull that in and check it into the repo like we did with country population? What is the desired final format? Same as the country format?

@gwincr11 yep i think it can be super simple -- just like
https://github.com/InstituteforDiseaseModeling/covasim/blob/master/covasim/data/country_age_distributions.py
except per state (i think that data is down to the county level, we don't need that, just by state). format doesn't have to match neher lab, can just add a separate loader to
https://github.com/InstituteforDiseaseModeling/covasim/blob/master/covasim/data/loaders.py
if 150 country age distribution is 40 kb then 50 states should be 15 kb so totally fine to commit directly to the repo

(btw, reason to include as python files is so it plays nicely with pip and doesn't require MANIFEST.in gymnastics -- can rethink that approach if/when it becomes necessary to)

Closed by @gwincr11 (#124 )


Allows multiple selection on panel "Go to file..."

IDEs like NetBeans allows to select multiple files on that panel. It's important because when I'm looking for a file, I probably need to open the main file and their complements.

Arrow right → ?


Data Grid Virtual Scrolling : Use a Sliding data window

What will this achieve?
Virtual scrolling would never run out of resources when used with extremely large datasets
How will success be measured?
When using an extremely large dataset (100k ,1M rows ...), you should be able to continue scrolling without overtaxing the browser resources.
Additional information
The way virtual scrolling works today is via 'bucket growth'. The fetch callback is called when it needs to grow the number of available rows. But it is a continual growth. If the dataset is extremely large, this will not be sustainable.  Supporting a sliding window approach would have the DataGrid asking the application to provide specific window of data based on the users scrolling direction.   The application would obtain and provide that data to the DataGrid

@crichtonibm Would you be able to contribute an enhancement for this request?

We've been investigating TanStack Table, a third-party, open-source offering, which provides extensive data table capabilities surpassing what our Carbon Datagrid offers. It provides much more flexibility and customization. TanStack Table is headless which means it can easily be added alongside Datagrid component in your product or application. The benefits of more flexibility for product teams and less maintenance for Carbon makes it a win win. Lastly, it is available in multiple frameworks including React and Web Component so it provides an option to non-React product teams.
For these reasons, we have decided to transition from building our own custom table component to using an example-based approach with TanStack Table. Datagrid will still exist in our library for existing teams but we are announcing the deprecation* of the Datagrid component in v2.54.0 release so teams can begin to work through the transition. Details about how to use both Datagrid and TanStack together can be found here.
*Deprecation means that no new features will be added however sev 1 and sev 2 bugs will be supported.


Tracking dependencies by artifact's creation time like GNU make

I'm sure we really need this..
It will provide ability to resume calculation based on availablity of artifacts and their creation time.
Without this feature kedro calculates entire pipeine from cratch.. And that is not suitable in many cases.

Hi @sergun thank you for opening the feature request. Could you please elaborate a bit more on what kind of feature you are looking for? There's a template for a feature request, and it would be great if you could fill them in.
What calculation would you like to know? The runtime of pipeline execution, or IO operation etc.

Description
Currently kedro can only calculate pipilenes from scratch and with some manually specified settings (you can specify nodes from which it can calculate everything or you can ask kedro to calculate only missing datasets by Runner.run_only_missing).
Context
In many use-cases model GNU make works better.. As you know make automaticlly tracks  dependencies between artefacts (the same does kedro) but it also automatically tracks artifacts' mtime and understand which artifact should be re-created by taking in mind mtime of output artefact and dependencies.
Such model is suitable e.g. for feature engineering, when you have a lot of SQL-scripts, each of them creates some temporary table. Tables are joined and finally you have some resulting table with feature values. It is nice that data scientist can modify some script and make will automatically understand that table which is created by this script should be re-created and all tables depend on it should be also re-calculated. And other tables should be untouched. This use-cases is completely unsupported by kedro.
Possible Implementation
No concreted ideas. But I think that we can integrate mtime in Dataset class and add ability to select nodes to be calculated based on mtime of input/output Datasets of each node.
Possible Alternatives
Do not see them.

As for me, it will be really great to add such a functionality!

I am not 100% sure that I follow what you are looking for, but my personal feeling so far is that this is a feature that can be achieved through the use of hooks.
All of the kedro DataSets are currently backed by fsspec, a quick scan of their API revealed that there is a .modified() method that returns a timestamp of the modified path.  I am not exactly sure how you would leverage this in a hook though.
You can access the kedro dataset instance dynamically by using getattr(catalog.datasets, 'dataset_name').  Inside the dataset instance, you will find all sorts of information about your dataset.  I was able to get to the modified time of my datasets, but it did not appear that it was leveraging fsspec for the file system agnostic methods.  Instead it seemed like it was specific to the filesystem type I was using.
Once you have the time what do you need to do with that?  I am not familiar with GNU make and how it utilizes mtime to avoid recreating artifacts that are not necessary to make.
Personally I feel like this would be another great application for #400.  If I could set a default update frequency (daily), and override that frequency in my catalog to tell kedro that this dataset is only refreshed (weekly/monthly/cron expression?).  Then it could figure out if its time to update or now.

Thanks @WaylonWalker  !
But I do not see how to skip execution of node's processing function from some hook.
The idea is to do not call this function if mtime of input datasets of a node are earlier than mtime of it's output datasets and these datasets exist.

I've quickly hacked together (with emphasis on hack) a prototype of what a hook would look like that enables something like this: https://gist.github.com/mzjp2/076bfd73b0215bda01ee71186966389d

That is a really cool hook @mzjp2! If we could tag nodes with a run frequency and combine with this it would make things easy to blindly run everything and only update out of date nodes.

Thanks @WaylonWalker !
But I do not see how to skip execution of node's processing function from some hook.
The idea is to do not call this function if mtime of input datasets of a node are earlier than mtime of it's output datasets and these datasets exist.

I see now.  What if you grabbed the ast or bytecode of the function of each node and cached it.  Then you can check if the function itself has changed since last run, or if input data has changed since the last run.

I found a parallel conversation is happening over on kedro.community.
https://discourse.kedro.community/t/speeding-up-pipeline-processing-with-change-detection/90

@WaylonWalker thanks for the link!
It is interesting that in our company we created very similar "in-house" solution based on old-school Makefiles :-)
With change detection based on MTIME of data files and processing scripts.
It works well but we decided to try kedro and wondered that there are no similar things implemented here..

@mzjp2 thanks a lot! Great job!
It seems that it should be core thing/concept in the future not hook based.. What do you think?

I think I'd like to get my hands dirty with this one. I'll look into this in the context of Hacktoberfest. I'll make a draft PR where I reference this issue and the discussion in kedro.community. Should be a fun one.
@dataengineerone @sergun FYI

@pascalwhoop I think it would be one of the most significant cotribution to kedro :1st_place_medal:
I think it make sense to consider both ways: hash-based / time-based  tracking of changes.
From my Makefile-based for ML experience I can say it is really cool when you do not need to think which nodes should be executed after some change (of params or data, or maybe code). BTW in the Makefile-based colution parameters were also files and they were incuded in make recipes as dependencies..
They only problematic place with time-based tracking is cases when you add whitespace or lineend symbol to a file with parameters or to source code script and make wants to recalculate something dependend on them :-)
I also like make becuase you identidy your task by artifact (filename) not by id of node.. I find this more intuitive..


Use a 8 bit-flag to store all avalonia property flags

What does the pull request do?
To reduce the memory cost by combining four flag fields into one, similar as WPF's.
What is the current behavior?
Every AvaloniaProperty needs four fields to store the flags.
public bool Inherits { get; private protected set; }
public bool IsAttached { get; private protected set; }
public bool IsDirect { get; private protected set; }
public bool IsReadOnly { get; private protected set; }

How was the solution implemented (if it's not obvious)?
[Flags]
private enum Flag : byte
{
    Inherits = 1,   // 0b_0000_0001
    IsAttached = 2, // 0b_0000_0010
    IsDirect = 4,   // 0b_0000_0100
    IsReadOnly = 8  // 0b_0000_1000
}
private Flag _flags;
public bool Inherits
{
    get => _flags.HasFlag(Flag.Inherits);
    private protected set => _flags = value ? (_flags | Flag.Inherits) : (_flags & ~Flag.Inherits);
}
public bool IsAttached
{
    get => _flags.HasFlag(Flag.IsAttached);
    private protected set => _flags = value ? (_flags | Flag.IsAttached) : (_flags & ~Flag.IsAttached);
}
public bool IsDirect
{
    get => _flags.HasFlag(Flag.IsDirect);
    private protected set => _flags = value ? (_flags | Flag.IsDirect) : (_flags & ~Flag.IsDirect);
}
public bool IsReadOnly
{
    get => _flags.HasFlag(Flag.IsReadOnly);
    private protected set => _flags = value ? (_flags | Flag.IsReadOnly) : (_flags & ~Flag.IsReadOnly);
}

I would prefer a bitwise check instead of HasFlag for performance reasons

Make sense, I've updated it accordingly.

Just out of curiosity. What is the memory gain? 3 bytes per object right? If so, it's aprox 3Mb each 1million objects... Or am I missing something due to memory alignment or something else?

I'd be interested in a performance comparison here as well. Would be interesting to see what the impact is.

I would prefer a bitwise check instead of HasFlag for performance reasons

This makes no difference as HasFlag can already be highly optimized by the JIT:
https://sharplab.io/#v2:EYLgxg9gTgpgtADwGwBYA0AXEBDAzgWwB8ABAJgEYBYAKBoDEAbbAcwAIB9AMyedwG4aNYBAgNWASQB2ACxhQAlhlwAKAJQ0A3jVY7WxAOwduLXADoAEnkYtl15qamyFS1QOoBfQdQDad3AF0aGEkAV3xWO1YQVmAATwwYTW1dRzlFXFYAXlZyNF0AenzWAAZgdmKK4vKKqmpdCVwAQQwMbDBZABMs1lI8wpKyyqqa4uSdcVwAEXlYMAxu9AKi0uqK8vIKsYaAJRhsDoB5SQZY7oAOHX6VofYNzY8gA=

@hez2010 Are you sure this is the case for all runtimes?
You can also use this: https://github.com/YohDeadfall/Avalonia/blob/master/src/Avalonia.Base/EnumExtensions.cs

I did some tests, the result turns out there's no memory difference so far, because of the memory aligment.
public class MockObject : AvaloniaObject
{
    public readonly AvaloniaProperty[] array = new AvaloniaProperty[30000];

    public void Setup()
    {
        for (int i = 0; i < 10000; i += 3)
        {
            array[i]     = AvaloniaProperty.Register<MockObject, string>("p" + i);
            array[i + 1] = AvaloniaProperty.RegisterAttached<MockObject, MockObject, string>("a" + i);
            array[i + 2] = AvaloniaProperty.RegisterDirect<MockObject, string>("d" + i, x => null);
        }
    }
}

var o = new MockObject();

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

var bytes1 = GC.GetTotalMemory(true);
Console.WriteLine("{0} bytes", bytes1);

o.Setup();

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

var bytes2 = GC.GetTotalMemory(true);
Console.WriteLine("{0} bytes, increase {1} Kb, {2} Mb", bytes2, (bytes2 - bytes1) / 1024.0, (bytes2 - bytes1) / 1024.0 / 1024.0);
Console.WriteLine("{0} bytes per instance", (bytes2 - bytes1) / (o.array.Length * 1.0));
GC.KeepAlive(o);

Both print on my machine (win11 x64)
329760 bytes
7547208 bytes, increase 7048.2890625 Kb, 6.883094787597656 Mb
240.5816 bytes per instance

But if one more flag added, you'll see the difference, bitwise takes a little less memory than boolean fields.
So, please feel free to close the PR.

You can see the memory layout using this tool:
https://github.com/SergeyTeplyakov/ObjectLayoutInspector
It might be worth to manually rearrange fields for better memory layout.

Either way, there shouldn't be many AvaloniaProperty objects in memory. Similar optimizations might make more sense for Visuals though. Or styled property values, but that part was already optimized this way iirc.


fix: only init TB once

When running multiple instances of the UI, you usually get an error like this:
1|mm  | [TELBOT] Error 409 Conflict: terminated by other getUpdates request; make sure that only one bot instance is running

This is due to the fact, that each time the module is startet, the bot is initialized, even if it has been initialized by the module of another UI instance before.
This change prevents an initialization if the bot is already created.

I have push you purpose in dev branch
Thx for your purpose
I will see if I have some change and I commit it in master


Ethernet support for STM32H747

Did I understand correctly from the description STM32H747 that the built-in Ethernet controller is not supported for STM32H747 in Zephyr?
If this is the case, then:

how difficult is it to support Ethernet on my own?
are you planning to support Ethernet in the near future?
if Ethernet is supported, will the IEEE1588/802.1 AS time synchronization subsystem with hardware timestamping be available?

@gmarull any advice for @i-and?

I haven't used Ethernet on H7 series, but it looks like F7 has support for it (see for example nucleo_f746zg board). In general existing drivers require little adjustments to function with other STM32 series, so unless H7 silicon has changed a lot, adding support for it may only consist on defining the peripheral in DT files and a few other small adjustments.

I haven't used Ethernet on H7 series, but it looks like F7 has support for it (see for example nucleo_f746zg board). In general existing drivers require little adjustments to function with other STM32 series, so unless H7 silicon has changed a lot, adding support for it may only consist on defining the peripheral in DT files and a few other small adjustments.

Additionally, I should mention that on this board, ethernet controller is not connected to MCU by default and you have to do some hw modifications to get it working. You'll get more information in the following post:
https://github.com/ARMmbed/mbed-os/pull/11827
By advance, sorry for the inconvenience.

To continue the conversation, please clarify: is Zephyr supported on two Cortex-M7 and Cortex-M4 cores of STM32H747, or on one of them? If both cores are supported, in which mode (AMP, SMP, ,...)?

@i-and Zephyr is supported on both cores in AMP. Though for now only a limited range of peripherals are supported. Also we still miss inter-core communication

@erwango Is it possible to use Hardware semaphore (HSEM) for peripheral ( ETH module etc. ) for synching between the cores?

@erwango Is it possible to use Hardware semaphore (HSEM) for peripheral ( ETH module etc. ) synching between the cores?

@Nukersson Depends on your need. If this is to protect usage between cores, we're currently trying to improve to existing code with https://github.com/zephyrproject-rtos/zephyr/pull/24862.
Does it answer your question ?

@erwango It completely does answering my question. Hopefully I can join to #24862 later, as I get more experience with Zephyr.

@erwango I am trying to use Ethernet on STM32H742VG.
I want to use Ethernet on H7 by referring to the source of F7 series. What sources of F7 should I reference?(dts, samples, ..)
I currently only know pinmux.c.
https://github.com/zephyrproject-rtos/zephyr/blob/master/boards/arm/nucleo_f756zg/pinmux.c#L37-L47
#ifdef CONFIG_ETH_STM32_HAL
	{ STM32_PIN_PC1, STM32F7_PINMUX_FUNC_PC1_ETH },
	{ STM32_PIN_PC4, STM32F7_PINMUX_FUNC_PC4_ETH },
	{ STM32_PIN_PC5, STM32F7_PINMUX_FUNC_PC5_ETH },
	{ STM32_PIN_PA1, STM32F7_PINMUX_FUNC_PA1_ETH },
	{ STM32_PIN_PA2, STM32F7_PINMUX_FUNC_PA2_ETH },
	{ STM32_PIN_PA7, STM32F7_PINMUX_FUNC_PA7_ETH },
	{ STM32_PIN_PG11, STM32F7_PINMUX_FUNC_PG11_ETH },
	{ STM32_PIN_PG13, STM32F7_PINMUX_FUNC_PG13_ETH },
	{ STM32_PIN_PB13, STM32F7_PINMUX_FUNC_PB13_ETH },
#endif  /* CONFIG_ETH_STM32_HAL */

I have a nucleo_h743zi board that I can test.
I am new to Ethernet on Zephyr. :sweat_smile:

I am trying to use Ethernet on the nucleo_h743zi board.
zephyr: https://github.com/KwonTae-young/zephyr/tree/stm32h743_ethernet
hal_stm32: https://github.com/KwonTae-young/hal_stm32
First of all, I have no experience using Ethernet in STM32. So I am very immature. :sweat_smile:
My first goal was the occurrence of a receive interrupt(HAL_ETH_RxCpltCallback()).
However, the current Ethernet function does not operate normally.
west init zephyrproject_stm32h743_ethernet -m https://github.com/KwonTae-young/zephyr --mr stm32h743_ethernet
cd zephyrproject_stm32h743_ethernet
west update
cd zephyr
source zephyr.sh
west build -b nucleo_h743zi samples/net/sockets/echo
west flash

1. Receive Buffer Unavailable(ETH_DMACSR)
Receive Buffer Unavailable occurs in ETH_DMACSR register when the driver under test is operating.

I don't know the exact reason, but changing the STM32 HAL Ethernet driver eliminated this symptom.
Bug in ETH_DMARxDescListInit - reported in the forum
diff --git a/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c b/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c
index a34d44e1cc9520032d42f46967b1a45157e768e1..29631ff647fc1be49cca4da1b01d2295c355c90e 100644
--- a/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c
+++ b/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c
@@ -2703,7 +2703,7 @@ static void ETH_DMARxDescListInit(ETH_HandleTypeDef *heth)
   WRITE_REG(heth->Instance->DMACRDLAR, (uint32_t) heth->Init.RxDesc);

   /* Set Receive Descriptor Tail pointer Address */
-  WRITE_REG(heth->Instance->DMACRDTPR, ((uint32_t)(heth->Init.RxDesc + (((uint32_t)(ETH_RX_DESC_CNT - 1))*sizeof(ETH_DMADescTypeDef)))));
+  WRITE_REG(heth->Instance->DMACRDTPR, ((uint32_t)(heth->Init.RxDesc) + ((uint32_t)(ETH_RX_DESC_CNT - 1))));
 }

 /**

before
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_iface_init: eth_iface_init() 625Line
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_stm32_hal_get_capabilities: eth_stm32_hal_get_capabilite
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x80
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x80
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0

after
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_iface_init: eth_iface_init() 625Line
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_stm32_hal_get_capabilities: eth_stm32_hal_get_capabilite
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0

2. Receive Queue Overflow Interrupt Status(ETH_MTLQICSR)
After resolving 1. Receive Buffer Unavailable(ETH_DMACSR), it becomes LINKUP and Receive Queue Overflow Interrupt Status((ETH_MTLQICSR)) occurs.

I am not sure how to solve this.
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: Link UP!
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x10000
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x10000
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0

The register of PHY chip is read well.
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: Basic Control Register: 0x3000
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: Basic Status Register: 0x7809
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: PHY Identifier 1 Register: 0x7
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: PHY Identifier 2 Register: 0xc131

Can someone please advise on this phenomenon? :confounded:

The RBU and other problems with ST's STM32H7 HAL ethernet driver are discussed and fixed at https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet.

The RBU and other problems with ST's STM32H7 HAL ethernet driver are discussed and fixed at https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet.

Would you like to implement the STM32H7's ETH driver for Zephyr?

Can’t get involved at this time.
From: Alexander Kozhinov<EMAIL_ADDRESS>Sent: Sunday, 31 May 2020 4:04 AM
To: zephyrproject-rtos/zephyr<EMAIL_ADDRESS>Cc: Alister Fisher<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Subject: Re: [zephyrproject-rtos/zephyr] Ethernet support for STM32H747 (#24767)
The RBU and other problems with ST's STM32H7 HAL ethernet driver are discussed and fixed at https://community.st.com/s/question/0D50X0000C6eNNSSQ2/bug-fixes-stm32h7-ethernet.
Would you like to implement the STM32H7's ETH driver for Zephyr?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/zephyrproject-rtos/zephyr/issues/24767#issuecomment-636364762, or unsubscribehttps://github.com/notifications/unsubscribe-auth/APXNYZP2GUMTI4KRXUQVEFLRUFDB5ANCNFSM4MS4IITQ.

With some help of STM32H745_Ethernet repository I was able to provide first working version of #26226 of stm32h7 ethernet driver tested on nucleo-h745zi-q dev board and with existing socket based zephyr samples. Corresponding issue #26174 were also opened.
Please feel free to provide your proposals to this PR.

I am trying to use Ethernet on the nucleo_h743zi board.
zephyr: https://github.com/KwonTae-young/zephyr/tree/stm32h743_ethernet
hal_stm32: https://github.com/KwonTae-young/hal_stm32
First of all, I have no experience using Ethernet in STM32. So I am very immature. 😅
My first goal was the occurrence of a receive interrupt(HAL_ETH_RxCpltCallback()).
However, the current Ethernet function does not operate normally.
west init zephyrproject_stm32h743_ethernet -m https://github.com/KwonTae-young/zephyr --mr stm32h743_ethernet
cd zephyrproject_stm32h743_ethernet
west update
cd zephyr
source zephyr.sh
west build -b nucleo_h743zi samples/net/sockets/echo
west flash

1. Receive Buffer Unavailable(ETH_DMACSR)
Receive Buffer Unavailable occurs in ETH_DMACSR register when the driver under test is operating.

I don't know the exact reason, but changing the STM32 HAL Ethernet driver eliminated this symptom.
Bug in ETH_DMARxDescListInit - reported in the forum
diff --git a/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c b/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c
index a34d44e1cc9520032d42f46967b1a45157e768e1..29631ff647fc1be49cca4da1b01d2295c355c90e 100644
--- a/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c
+++ b/stm32cube/stm32h7xx/drivers/src/stm32h7xx_hal_eth.c
@@ -2703,7 +2703,7 @@ static void ETH_DMARxDescListInit(ETH_HandleTypeDef *heth)
   WRITE_REG(heth->Instance->DMACRDLAR, (uint32_t) heth->Init.RxDesc);

   /* Set Receive Descriptor Tail pointer Address */
-  WRITE_REG(heth->Instance->DMACRDTPR, ((uint32_t)(heth->Init.RxDesc + (((uint32_t)(ETH_RX_DESC_CNT - 1))*sizeof(ETH_DMADescTypeDef)))));
+  WRITE_REG(heth->Instance->DMACRDTPR, ((uint32_t)(heth->Init.RxDesc) + ((uint32_t)(ETH_RX_DESC_CNT - 1))));
 }

 /**

before
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_iface_init: eth_iface_init() 625Line
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_stm32_hal_get_capabilities: eth_stm32_hal_get_capabilite
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x80
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x80
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0

after
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_iface_init: eth_iface_init() 625Line
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_stm32_hal_get_capabilities: eth_stm32_hal_get_capabilite
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:00.500,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:01.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0

2. Receive Queue Overflow Interrupt Status(ETH_MTLQICSR)
After resolving 1. Receive Buffer Unavailable(ETH_DMACSR), it becomes LINKUP and Receive Queue Overflow Interrupt Status((ETH_MTLQICSR)) occurs.

I am not sure how to solve this.
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: Link UP!
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:02.000,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:02.500,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x10000
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0
[00:00:03.000,000] <dbg> eth_stm32_hal.rx_thread:
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_DMACSR: 0x0
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MTLQICSR: 0x10000
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACPCSR: 0x0
[00:00:03.500,000] <dbg> eth_stm32_hal.rx_thread: ETH_MACLCSR: 0x0

The register of PHY chip is read well.
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: Basic Control Register: 0x3000
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: Basic Status Register: 0x7809
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: PHY Identifier 1 Register: 0x7
[00:00:00.000,000] <dbg> eth_stm32_hal.eth_initialize: PHY Identifier 2 Register: 0xc131

Can someone please advise on this phenomenon? 😖

I've tested your development branch for the driver some weeks ago and found the problem in dma_tx/rx buffers allocation. I've also got same issue (RBU/TBU). Since the buffers have been moved to non-cachable SRAM area (AXI-SRAM, AHB-SRAM), which is accessible by the ETH-DMA, the driver works fine with sample projects like samples/net/sockets/dumb_http_server_mt. I've also created according PR (see above).

@lochej @erwango @ABOSTM
Dear all I've tested the interrupt approach (see comments in #26226) from @lochej on my nucleo-h745zi-q board. The branch itself is in my repo and is ready for @lochej to be integrated after the current PR is merged.
The performance were tested with following ping configuration on 1k ping packets with 0.01 sec. ping interval:
ping -i 0.01 <IP_ADDRESS> -c 1000 > log_file.log
The board has static IP address and were connected to my FritzBox 7590 router. My Mac laptop were connected over 5GHz WiFi to the same router. The distance between the router and the laptop was just about 1 meter.
Note: Due to used WiFi my performance tests may vary from @lochej ones and can face lost packets.
The results of the tests are the following:

Old approach: Polling TX:
1000 packets transmitted, 133 packets received, +2 duplicates, 86.7% packet loss round-trip min/avg/max/stddev = 1.037/347.861/1002.657/372.646 ms

New approach (thnx to @lochej ): Interrupt TX:
--- <IP_ADDRESS> ping statistics --- 1000 packets transmitted, 508 packets received, 49.2% packet loss round-trip min/avg/max/stddev = 1.154/65.999/742.486/150.473 ms

Thus we see the increase in performance of factor ~2 (372ms / 150ms and 86.7%/49.2%) in ping response deviation time and number of the lost packets. Especially interesting is the average ping response time decrease by almost factor of ~5.3 = avg(348ms) / avg(66 ms). Thus the gain of the improvement can be clearly seen and the improvement from @lochej must be implemented.
P.S.: @lochej Please feel free to use my repo to create your own PR, since this my branch is only copy/paste of your solution.

@lochej @erwango @ABOSTM
Dear all I've tested the interrupt approach (see comments in #26226) from @lochej on my nucleo-h745zi-q board. The branch itself is in my repo and is ready for @lochej to be integrated after the current PR is merged.
The performance were tested with following ping configuration on 1k ping packets with 0.01 sec. ping interval:
ping -i 0.01 <my_ipv4> -c 1000 > log_file.log
The board has static IP address and were connected to my FritzBox 7590 router. My Mac laptop were connected over 5GHz WiFi to the same router. The distance between the router and the laptop was just about 1 meter.
Note: Due to used WiFi my performance tests may vary from @lochej ones and can face lost packets.
The results of the tests are the following:
1. Old approach: Polling TX:
   `1000 packets transmitted, 133 packets received, +2 duplicates, 86.7% packet loss round-trip min/avg/max/stddev = 1.037/347.861/1002.657/372.646 ms`

2. New approach (thnx to @lochej ): Interrupt TX:
   `1000 packets transmitted, 508 packets received, 49.2% packet loss round-trip min/avg/max/stddev = 1.154/65.999/742.486/150.473 ms`

Thus we see the increase in performance by factor ~2 (372ms / 150ms and 86.7%/49.2%) in ping response deviation time and number of the lost packets. Especially interesting is the average ping response time decrease by almost factor of ~5.3 = avg(348ms) / avg(66 ms). Thus the gain of the improvement can be clearly seen and the improvement from @lochej must be implemented.
P.S.: @lochej Please feel free to use my repo to create your own PR, since this my branch is only copy/paste of your solution.

Thanks for your feedback on my method :) Sure that will be a great addition to the ethernet driver for both performance et reliability. I will need a bit of time to PR my method due to vacations.
To complete your benchmarks:
I didn't notice packet losses when connecting to the board (Nucleo-H743ZI) directly or when passing through my router to connect to the board (with DHCPv4 functionality working perfectly).
Also my ping response time where excellent with very low stddev and avg response time of less than 5ms even when passing through wifi to the router down to the board. Note that my system clock was 96MHz.
I will provide more benchmarks in the future and additionnal support for the nucleo-h743zi board especially for the ethernet and system clocks support.

@lochej @erwango @ABOSTM
Dear all I've tested the interrupt approach (see comments in #26226) from @lochej on my nucleo-h745zi-q board. The branch itself is in my repo and is ready for @lochej to be integrated after the current PR is merged.
The performance were tested with following ping configuration on 1k ping packets with 0.01 sec. ping interval:
ping -i 0.01 <my_ipv4> -c 1000 > log_file.log
The board has static IP address and were connected to my FritzBox 7590 router. My Mac laptop were connected over 5GHz WiFi to the same router. The distance between the router and the laptop was just about 1 meter.
Note: Due to used WiFi my performance tests may vary from @lochej ones and can face lost packets.
The results of the tests are the following:
1. Old approach: Polling TX:
   `1000 packets transmitted, 133 packets received, +2 duplicates, 86.7% packet loss round-trip min/avg/max/stddev = 1.037/347.861/1002.657/372.646 ms`

2. New approach (thnx to @lochej ): Interrupt TX:
   `1000 packets transmitted, 508 packets received, 49.2% packet loss round-trip min/avg/max/stddev = 1.154/65.999/742.486/150.473 ms`

Thus we see the increase in performance by factor ~2 (372ms / 150ms and 86.7%/49.2%) in ping response deviation time and number of the lost packets. Especially interesting is the average ping response time decrease by almost factor of ~5.3 = avg(348ms) / avg(66 ms). Thus the gain of the improvement can be clearly seen and the improvement from @lochej must be implemented.
P.S.: @lochej Please feel free to use my repo to create your own PR, since this my branch is only copy/paste of your solution.

Thanks for your feedback on my method :) Sure that will be a great addition to the ethernet driver for both performance et reliability. I will need a bit of time to PR my method due to vacations.
To complete your benchmarks:
I didn't notice packet losses when connecting to the board (Nucleo-H743ZI) directly or when passing through my router to connect to the board (with DHCPv4 functionality working perfectly).
Also my ping response time where excellent with very low stddev and avg response time of less than 5ms even when passing through wifi to the router down to the board. Note that my system clock was 96MHz.
I will provide more benchmarks in the future and additionnal support for the nucleo-h743zi board especially for the ethernet and system clocks support.

Which project have you used to ping the board for the performance tests? I've used dumb_http_server_mt from Zephyr's examples pool.

@lochej @erwango @ABOSTM
Dear all I've tested the interrupt approach (see comments in #26226) from @lochej on my nucleo-h745zi-q board. The branch itself is in my repo and is ready for @lochej to be integrated after the current PR is merged.
The performance were tested with following ping configuration on 1k ping packets with 0.01 sec. ping interval:
ping -i 0.01 <my_ipv4> -c 1000 > log_file.log
The board has static IP address and were connected to my FritzBox 7590 router. My Mac laptop were connected over 5GHz WiFi to the same router. The distance between the router and the laptop was just about 1 meter.
Note: Due to used WiFi my performance tests may vary from @lochej ones and can face lost packets.
The results of the tests are the following:
1. Old approach: Polling TX:
   `1000 packets transmitted, 133 packets received, +2 duplicates, 86.7% packet loss round-trip min/avg/max/stddev = 1.037/347.861/1002.657/372.646 ms`

2. New approach (thnx to @lochej ): Interrupt TX:
   `1000 packets transmitted, 508 packets received, 49.2% packet loss round-trip min/avg/max/stddev = 1.154/65.999/742.486/150.473 ms`

Thus we see the increase in performance by factor ~2 (372ms / 150ms and 86.7%/49.2%) in ping response deviation time and number of the lost packets. Especially interesting is the average ping response time decrease by almost factor of ~5.3 = avg(348ms) / avg(66 ms). Thus the gain of the improvement can be clearly seen and the improvement from @lochej must be implemented.
P.S.: @lochej Please feel free to use my repo to create your own PR, since this my branch is only copy/paste of your solution.

Thanks for your feedback on my method :) Sure that will be a great addition to the ethernet driver for both performance et reliability. I will need a bit of time to PR my method due to vacations.
To complete your benchmarks:
I didn't notice packet losses when connecting to the board (Nucleo-H743ZI) directly or when passing through my router to connect to the board (with DHCPv4 functionality working perfectly).
Also my ping response time where excellent with very low stddev and avg response time of less than 5ms even when passing through wifi to the router down to the board. Note that my system clock was 96MHz.
I will provide more benchmarks in the future and additionnal support for the nucleo-h743zi board especially for the ethernet and system clocks support.

Which project have you used to ping the board for the performance tests? I've used dumb_http_server_mt from Zephyr's examples pool.

I have used the simple Dumb HTTP

@lochej Just trieb to ping with 1ms and 500us. It won't work properly on my machine. I'll try it later (at the time you PR is prepared) with proper router settings or even direct connection to my laptop.

@carlescufi Actually solved by #26226. Should it be closed now?

Fixed by #26226


Question - TFRecords and contiguous examples

Hi,
From the documentation:

WARNING: This samplers assume that classes examples are contigious,
at least enough that you can get example_per_class numbers
of them consecutively. This requirements is needed to make the
sampling efficient and makes dataset constuctionn oftentime easier as
there is no need to worry about shuffling. Somewhat contigious means
its fine to have the same class in multiples shards as long as the
examples for the same classes are contigious in that shard.

Most likely I am confused with the definition of shards here.
For my given experiment, I have 12 classes. For every class, I have its corresponding tfrecord file. In other words, examples for a class are contiguous since a tfrecord file contains example of a particular class.
I used TFRecordDatasetSampler with default parameters (i.e. example_per_class = 2, batch_size = 32) but I do not get the representatives from all classes.
# example output
tf.Tensor(
[10 10 10 10 10 10 10 10 10  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1
 11 11 11 11 11 11 11 11], shape=(32,), dtype=int64)

Would appreciate it if you could help provide some insight into how one should create the tfrecord files to utilize this sampler.
Regards & thanks
Kapil

Hi Kapil, I just pushed some updates to the TFRecordDatasetSampler and they should be available on pypi now. The sampler should now work as expected if each class is a separate tfrecord file. Please let me know if you still run into issues though.

quick update to say I'm going to try and look at this later this week.

So I've been working on this and the TFRecordDatasetSampler is a little tricky. Ideally we would like to pass a set of TFRecord datasets to choose_from_datasets, but this turns out to be pretty slow. Instead we use the interleave to randomly choose cycle_length worth of tf record files from disk and read block_length number of examples from each tf record iterator. If we make each tf record a single class, this makes it easy to ensure that a batch contains cycle_length worth of classes with block_length number of examples per class.
However, this assumes that the number of tf records in each file is a multiple of block_length and that our batch_size is equal to cycle_length * block_length. If one of the iterators is shorter it will run out of examples early and another tf record will be loaded. If that happens, then the subsequent batches will be offset. For example:
tfA = [0,0,0,0]
tfB = [1,1,1]  # <- this iterator is short one item
tfC = [2,2,2,2]
cycle_length = 2
block_length = 2

batch_1 = [0,0,1,1] # first batch is fine
batch_2 = [0,0,1,2] # tfB runs out so we load tfC
batch_3 = [2,2,2,1] # now all subsequent batches are offset

The other issue is that each tf record is loaded as an iterator, meaning that there is no way to shuffle the items within the record, meaning we will always consume the record examples in the same order. Additionally, if the iterator is longer than block_length we will need to exhaust the iterator before we are able to load a new class.
The current solution that I'm using is to write multiple tf records per class where I randomly sample the elements and make the size equal to block_length. This ensures that each tf record is a random sample of examples from the class and that the tf record is exhausted after a single batch. So the previous example would then be something like.
tfA_0 = [0,0]
tfA_1 = [0,0]
tfB_0 = [1,1]
tfB_1 = [1,1]
tfC_0 = [2,2]
tfC_1 = [2,2]
cycle_length = 2
block_length = 2

batch_1 = [0,0,1,1]  # first batch is fine tfA_0, tfB_1
batch_2 = [2,2,1,1] # now we don't need to finish the first iterator and can load new classes tfC_1, tfB_1
batch_3 = [2,2,0,0] # and the offsets are now gone tfC_0, tfA_1

Regarding the infinite dataset, we currently have a repeat added after the deserialization so the dataset will repeat forever. I'll look at adding a parameter to set the number of repeats as part of the next release.

Support for setting the number of repeats was just added in PR #196

@owenvallis
Hello, regarding this ,

WARNING: This sampler requires that each TF Record file contain
contiguous blocks of classes where the size of each block is a multiple
of example_per_class.

Here is a imagenet dataset in tfrecord format. I'm not confident if these tfrecord contain contiguous blocks of classes. In this cases, how should I approach on this tfrecord dataset with tf-similarity tfrecord-sampler? Any tips?

Also, unlike TFDatasetMultiShotMemorySampler, the TFRecordDatasetSampler doesn't have any positional paramo for augmentation. Shouldn't it be included?

I need to adopt the following approach, but the problem is that I don't have control over the occurrence of number of each class per batch.
def dataset_for_class(i):
    i = tf.cast(i, tf.int32)
    return dataset.filter(lambda label: label == i)

dataset = [1,2,3,1,2,3,1,1,4,5,1,1,5,1,5,4]
len_ds = len(dataset)

dataset = tf.data.Dataset.from_tensor_slices(dataset)
dataset = tf.data.Dataset.range(len_ds).interleave(
    dataset_for_class,
    cycle_length=1,
)
dataset = dataset.prefetch(tf.data.AUTOTUNE)
list(dataset.as_numpy_iterator())

[1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 5]

However, the expected tfrecord format described here is not always smooth to achieve. For example, an existing tfrecord format which I may want to use for similarity modeling, need to first desterilize and modify the class order accordingly and save it again as per needed.


⚠ Bofrost.lu has degraded performance

In 177b2f3, Bofrost.lu (https://www.bofrost.lu) experienced degraded performance:

HTTP code: 200
Response time: 3074 ms

Resolved: Bofrost.lu performance has improved in f0c1284 after 19 minutes.


karma needs to be at least 0.12.8

Fixes issue #2 and #3

Works for me as well

Yar.. sorry another one I missed. Also will be merged with https://github.com/rendrjs/rendr-examples/pull/8 for 00 and the rest will follow.


Flashing Asus TUF Gaming FX505GT (FX505GT-BI5N7)

@pexcfequinnet

Motherboard model: TUF Gaming FX505GT (FX505GT-BI5N7)
Motherboard chipset: HM370
CPU Model: Intel Core i5-9300H
Graphic card: ASUS GTX 1650 Mobile
GPU PCI VendorID:DeviceID: 10DE:1F91
GPU PCI subsystem IDs: 1043:1AB1
VRAM size: 4096MB (4GB)
New BAR size (GPU-Z): 4096MB (4GB)
New BAR size (nvidia-smi): 4096MB
NVIDIA driver version: 546.33

Excuse me? how do you enable rebar? im on AMD variant and everytime i want to flash bios it says selected file is not a proper bios.
FX505DU
Ryzen 7 3750H
GTX 1660Ti Mobile

Originally posted by @merlinch in https://github.com/terminatorul/NvStrapsReBar/issues/1#issuecomment-1990842018

@pexcfequinnet Can you still provide information on how you flashed the board ?
@merlinch people report success using afudos.exe or afuwin.exe for flashing the firmware, on an USB stick with FreeDOS installed. Although I understand of laptops it just doesn't work without a (cheap) hardware flasher
If you try afudos, make sure you have a good battery level, as I understand Asus laptops do not have dual BIOS or flashback, for BIOS recovery.

Refering to #41

After trying all possible methods, I finally came to the conclusion that on Asus laptops where there is no support for the Asus AI suite, the only way to flash the modded BIOS is with a programmer CH341A

so maybe he use bios flasher

@pexcfequinnet

Motherboard model: TUF Gaming FX505GT (FX505GT-BI5N7)
Motherboard chipset: HM370
CPU Model: Intel Core i5-9300H
Graphic card: ASUS GTX 1650 Mobile
GPU PCI VendorID:DeviceID: 10DE:1F91
GPU PCI subsystem IDs: 1043:1AB1
VRAM size: 4096MB (4GB)
New BAR size (GPU-Z): 4096MB (4GB)
New BAR size (nvidia-smi): 4096MB
NVIDIA driver version: 546.33

Excuse me? how do you enable rebar? im on AMD variant and everytime i want to flash bios it says selected file is not a proper bios. FX505DU Ryzen 7 3750H GTX 1660Ti Mobile
Originally posted by @merlinch in #1 (comment)

Since I have an Intel variant of this laptop, I can use Intel Flash Programming Tool to flash the bios.

@pexcfequinnet Hi, How did you solve the problem of the lack of "Above 4G Decoding"? The DSDT Patching can't be done according to my attempts.
On the "FX506LH"(Mine), "Above 4G Decoding" is present but can't be enabled. "Above 4G MMIO BIOS assignment" is present and can be enabled.
On the "FX505GT" BIOS, "Above 4G Decoding" is completely missing. "Above 4G MMIO BIOS assignment" is the same as in "FX506LH". So I think both laptops are almost the same.

@pexcfequinnet Hi, How did you solve the problem of the lack of "Above 4G Decoding"? The DSDT Patching can't be done according to my attempts in "FX505GT" and "FX506LH" BIOS.
On the "FX506LH"(Mine), "Above 4G Decoding" is present but can't be enabled. "Above 4G MMIO BIOS assignment" is present and can be enabled. On the "FX505GT" BIOS, "Above 4G Decoding" is completely missing. "Above 4G MMIO BIOS assignment" is the same as in "FX506LH". So I think both laptops are almost in the same situation.

@Bl4ckfrost Looking at the the image you uploaded here, the "Above 4G Decoding" in your bios seems to be a header type so no value can be changed there. As for the "Above 4G MMIO BIOS assignment", enable that solve the problem for 4G decoding in GPU-Z.

@pexcfequinnet I already did the "Adding FFS module" step.
In the "Using UEFIPatch" step, UEFIPatch say "No patches can be applied to input file", so skip to "DSDT Patching".
In "DSDT Patching", now "Above 4G Decode" is enabled in GPU-Z, so according to my understanding "DSDT Patching" is not needed.
The mod bios and the original have identical padding, that means everything is fine.
So next step is disable the BIOS Lock with Grub and flash the modded BIOS with FPT, I am right?

@pexcfequinnet I already did the "Adding FFS module" step. In the "Using UEFIPatch" step, UEFIPatch say "No patches can be applied to input file", so skip to "DSDT Patching". In "DSDT Patching", now "Above 4G Decode" is enabled in GPU-Z, so according to my understanding "DSDT Patching" is not required. The mod bios and the original have identical padding, that means everything is fine. So next step is disable the BIOS Lock with Grub and flash the modded BIOS with FPT, I am right?

@Bl4ckfrost Yes. Before you flash the modded bios you should dump the original bios first to verify the size of the modded bios with the original one. If size is matching with original one (e.g 16384kb to 16384kb), you are ready to flash the modded bios.

@Bl4ckfrost Yes. Before you flash the modded bios you should dump the original bios first to verify the size of the modded bios with the original one. If size is matching with original one (e.g 16384kb to 16384kb), you are ready to flash the modded bios.

@pexcfequinnet I used the BIOS from ASUS page for the mod. If I remove the capsule header, the sizes mach perfectly. So, "asusmodnocapsule.bin" is the one to flash.

@Bl4ckfrost Yes. Before you flash the modded bios you should dump the original bios first to verify the size of the modded bios with the original one. If size is matching with original one (e.g 16384kb to 16384kb), you are ready to flash the modded bios.

@pexcfequinnet I used the BIOS from ASUS page for the mod. If I remove the capsule header, the sizes mach perfectly. So, "asusmodnocapsule.bin" is the one to flash.

@Bl4ckfrost I'd recommend you to use your dumped bios to mod instead, since the asus bios on their website often does not contain any information specific to your laptop like serial number and mac address, but your mileage may vary.

@pexcfequinnet Is this normal? I put "n" for now.

@pexcfequinnet Is this normal? I put "n" for now.

@Bl4ckfrost your modded bios does not contain Intel ME. Try modding your dumped bios and flash it again.

@pexcfequinnet Is this normal? I put "n" for now.

@Bl4ckfrost Seems like something is wrong with your ME region. I'd say try dumping only the bios region using fptw64 -bios -d <yourfilename.bin> and then try modding again to see if the error presist.

@pexcfequinnet It worked lol. Thanks for helping me!
I flashed the new modded dump using "fptw64 -bios -f <filename.bin>" and and it worked perfectly.

@pexcfequinnet It worked lol. Thanks for helping me! I flashed the new modded dump using "fptw64 -bios -f <filename.bin>" and and it worked perfectly.

Nice job! Be sure to post on #1 to let the creator know.

@pexcfequinnet It worked lol. Thanks for helping me! I flashed the new modded dump using "fptw64 -bios -f <filename.bin>" and and it worked perfectly.

@Bl4ckfrost Nice job! Be sure to post on #1 to let the creator know.

Bro, maybe u can help me? After flashing modded bios on asus laptop and enabling ReBar, get BSODS "VIDEO MEMORY INTERNAL ERROR"
Ryzen 7 4800h
RTX 2060
Asus FX506IV

@pexcfequinnet It worked lol. Thanks for helping me! I flashed the new modded dump using "fptw64 -bios -f <filename.bin>" and and it worked perfectly.

@Bl4ckfrost Nice job! Be sure to post on #1 to let the creator know.

Bro, maybe u can help me? After flashing modded bios on asus laptop and enabling ReBar, get BSODS "VIDEO MEMORY INTERNAL ERROR"
Ryzen 7 4800h RTX 2060 Asus FX506IV

@Tykwha other than trying to build the binary file yourself and check for any wrong value if you're using the released one, I can't really help much because I don't know much on AMD platform.

@pexcfequinnet I followed the same steps with my laptop but I have a problem, see #22
It looks like we have the same platform.
What is your BIOS version?
NvStrapsRebar version ?
I added the FFS after the last module of the volume (as in the guide), did you do the same?

@pexcfequinnet I followed the same steps with my laptop but I have a problem, see #79 It looks like we have the same platform. What is your BIOS version? NvStrapsRebar version ? I added the FFS after the last module of the volume (as in the guide), did you do the same?

@Uranium2010 I'm currently on version 305, modded with the latest version of NvStrapRebar.
As for the FFS module, I followed exactly what the instructions told me to do:

Adding the module to the end of the volume.
Skipping DSDT as it is not needed (any attempt at patching DSDT give me errors).

Hello, i have the same laptop and Ive been considering enabling rebar. Do you notice any performance boost? Is there any special instruction I'd need to follow or could I use the 'normal' procedure?
Thanks in advance.


feat: Add 'default-branch' support.

@telackey bother you to rebase?

@telackey just a friendly nudge, mind rebasing 👍

@telackey closing now due to inactivity, would love to land this if you can rebase and sign the CLA 👍
appreciate the contribution.


clinfo Number of plataforms1 (only pocl) does not list vc4cl

Hello, first of all, thank you for the project. I'm using the raspberrypi model B v1.2 board and debian bookworm (raspberrypi OS in Imager). I was able to successfully compile the VC4CLStdLib projects,
VC4C and VC4CL. But when listing the clinfo it doesn't appear as it should, in v3d_info I can get the information

Am I forgetting to do something?

https://github.com/doe300/VC4CL/issues/110


Add Cache Path to Lleaves Prediction Code to prevent a re-compilation

As per the discussions with Prof @jiazou-bigdata, Lleaves compilation takes extremely long time for 500 & 1600 trees, hence skipping the compilation during benchmark and instead using the Cache to load pre-compiled Model from memory.

Looks good! Thanks.


Make Products Changeset Aware

API PR: https://github.com/SeedCompany/cord-api-v3/pull/2471
┆Issue is synchronized with this Monday item by Unito
┆Link To Item: https://seed-company-squad.monday.com/boards/3451697530/pulses/3466494554

Having a little styling problems with the badge outline not stretching to the icon when a prop is changed...
Not sure if this is part of the scope, but the languageEngagement where the products are added to doesn't have a changeset badge with changed mode.
Also not sure what to do with the title for a Product with Other type. Should that also have a changeset badge? should it be even changeset aware in the first place?

Also note that I put icons on the right instead of the left. I thought it went better with the flow of how the props are displayed, instead of changing margins on a changeset (the icon overlaps the label when it's on the left.)

This lived to the end of its lifetime and died


Clarify behaviour of scss with malformed bounds

This clarifies that malformed capabilities have bounds of [0,0) for the purposes of SCSS. I think their bounds don't affect any other instructions except GCBASE and GCLEN which already specify 0. CBLD is unaffect because it checks for malformed capabilities explicitly.
This behaviour can almost be inferred from GCBASE and GCLEN, but I think it's better to be explicit rather than to depend on the assumption that SCSS behaves the same as those instructions.
Fixes #281

late in adding this, but I think from a HW perspective it's much better to always return false when either cs1 or cs2 is malformed.
Currently, it looks like

isMalformed(cs1)
isMalformed(cs2)
scss

0
0
cs2.xbounds <= cs1.xbounds

0
1
true

1
0
cs2.xbounds == 0

1
1
true

It's the case that cs1 is malformed while cs2 is not which adds extra logic.  In this case, we need to check that the bounds of cs2==0 for it to have bounds that are equal or lesser.  It's much easier to just say, that if the bounds are malformed it cannot be a subset - since we are arbitrarily deciding a standardization.

Do you think all hardware would be implemented like that? I mean I would have thought you just have some kind of "bounds decoder" unit and then always do the comparison, so your SCSS unit doesn't actually care about whether the bounds are malformed or not (basically how the Sail code works).
That "bounds decoder" would already output 0s for malformed caps because that's what GCBASE/GCLEN require.
(I'm not a hardware designer though so maybe it's not that simple?)

It's certainly simpler to say "just return zero" because that way it's just a single AND gate on the result if either input is malformed.
It's better if the spec doesnt push additional logic into the calculation, and forcing bounds to zero after calculation and before the comparison is a waste of logic delay.
So I'm in favour of simplifying it. Cheaper hardware, and a defined output.

Sounds reasonable. The only issue I can imagine is that it's a bit inconsistent with gcbase and gclen - if software wanted to calculate scss using those then it would be inconsistent.  I dunno if you'd really want to do that tbf but it might be a bit of a paper cut? Presumably malformed caps are ideally rare?

Eh, on the architectural consistency front this could go either way: you might expect that SCSS is consistent with CBLD, for example.

That's a fair point

While we're at it, are there any other instructions which need additional definition for malformed inputs? It would be good to fix them all at once

Going through all the Zcheripurecap instructions:
7.1.1. CMV Either detag or leave the tag (see note below)
7.1.2. MODESW N/A: shouldn't be able to have a malformed cap as PCC
7.1.3. CADDI Should probably clear the tag, as it can't meaningfully do a representability check
7.1.4. CADD As CADDI
7.1.5. SCADDR As CADDI
7.1.6. ACPERM Doesn't touch bounds and legalises permissions, so same as CMV
7.1.7. SCMODE As CMV
7.1.8. SCHI Always clears the tag, so fine.
7.1.9. SCEQ Needs to be a proper bit-compare, so don't do anything special for malformed
7.1.10. SENTRY As CMV
7.1.11. SCSS As we've discussed here, go for "always return false" if either is malformed?
7.1.12. CBLD Can't set the tag on malformed things
7.1.13. GCTAG The current spec wording would imply this should return 0 on malformed inputs.
7.1.14. GCPERM Just extracts permissions. Gives 0 on malformed permissions
7.1.15. GCHI Needs to just extract the bits, nothing special for malformed
7.1.16. GCBASE Gives 0 on malformed bounds
7.1.17. GCLEN Gives 0 on malformed bounds
7.1.18. SCBNDSI Subset check should fail if input bounds malformed, tag needs to be cleared if producing a malformed cap, which is now possible
7.1.19. SCBNDS Same as SCBNDSI
7.1.20. SCBNDSR Same as SCBNDSI
7.1.21. CRAM Nothing relevant
7.1.22. LC Shouldn't allow loads via malformed caps, could clear the tag on loading a malformed cap.
7.1.23. SC Shouldn't allow stores via malformed caps, could clear the tag on storing a malformed cap.
From other instructions, it's just jumps that seem interesting: Shouldn't be allowed to malformed caps
It's important to make sure CBLD and SCBNDS* (plus similarly ACPERM) refuse to produce tagged malformed caps, which we've now addressed: these are the only way to actually create a tagged one architecturally. That should just leave LC as a way for one to enter the processor.
Then just SCSS, GCLEN, GCBASE, GCPERM, SCMODE have special cases that affect how they present the values other than just for the tag. I think we've addressed all of these well now.
To cover the rest of the instructions, I can then see four different reasonable strategies:

Assume it never happens and don't worry about it. If memory can bit flip or get you uninitialised, potentially tagged values, you're already in trouble.
Always do the malformed check whenever you're writing back a cap, anding the result with the tag. This is presumably tricky on lc in terms of timing.
Always do a malformed check when reading from the register file and clear the tag input to other things if it fails. I imagine this is reasonable: the hardware has to be there anyway, and the tag shouldn't be a timing-critical input into other things. Might be bad for power though...
(Slightly different to 3, might be better for power, but more confusing to get right). Do the malformed check wherever the tag can be observed only, so on GCTag, as the store data, and when being used to authorise operations, but not on CMV and friends.

Architecturally, I think 2, 3 and 4 are indistinguishable, so the spec doesn't have to clarify. 1 is observably different from the others only if you're worried about what to do if the memory can contain things that weren't written by the core.
All that to say, I don't think we need extra clarification, though the current wording of "Capabilities with malformed bounds are always invalid anywhere in the system i.e. their tags are always 0." could be read to mean 1 (i.e. it's telling you they can't exist tagged so the core doesn't have to worry about it), or (2, 3 or 4), i.e. this is something the design should make sure is true.

Just a quick comment, although we need to go through them all thoroughly.
Making CMV change the output prevents an out-of-order CPU from just increasing the reference count of the source physical register instead of actually copying the data, so would have performance implications.
All the others actually require execution, although modifying load data is undesirable (although we already plan to tag clear on LC on some occasions - but that's not dependant on the loaded data).

Good point! Sounds like 4. may be the way to go in practice then.

maybe - although that adds more data dependant logic to stores.
how about 4 but not on stores? would we want memcpy to change data?

"Capabilities with malformed bounds are always invalid anywhere in the system i.e. their tags are always 0." should probably be changed to something like "Bounds checks always fail if the authorising capability has malformed bounds. The core will never construct a tagged capability with malformed bounds."

Some thoughts:
We need to define:

all bounds calculate as zero for malformed tagged caps: GCBASE, GCLEN, SCBNDS*  (for SCBNDS* it has the result of detagging the result)
some instructions as propagating malformed tagged caps with no modification: CMV, LC, SC
some instructions as detagging malformed tagged caps: CADDI, CADD, SCADDR
some instructions as propagating malformed tagged caps, even though there is a metadata update: ACPERM, SCMODE, SENTRY
all get instructions other than in point 1 above don't both with the malformed check i.e. GCTAG, GCPERM, GCHI
SCEQ does a full bitwise compare regardless

I think that covers it....

some instructions as detagging malformed tagged caps: CADDI, CADD, SCADDR

I think this is not necessary because the address of a malformed cap doesn't matter since its bounds are 0?

Some thoughts:
We need to define:

all bounds calculate as zero for malformed tagged caps: GCBASE, GCLEN, SCBNDS*  (for SCBNDS* it has the result of detagging the result)
some instructions as propagating malformed tagged caps with no modification: CMV, LC, SC
some instructions as detagging malformed tagged caps: CADDI, CADD, SCADDR
some instructions as propagating malformed tagged caps, even though there is a metadata update: ACPERM, SCMODE, SENTRY
all get instructions other than in point 1 above don't both with the malformed check i.e. GCTAG, GCPERM, GCHI
SCEQ does a full bitwise compare regardless

I think that covers it....

I agree that should be the effect. I don't think this needs to litter the spec too much though? Hopefully we can specify that malformed caps fail all bounds and representability checks, and people can look at the sail to make fully sure? The current notes on GCBASE and GCLEN plus the proposed note on SCSS should be the only cases that need to address it explicitly?

I would say that the page for every instruction which writes a capability should have a malformed cap handling statement so it's unambiguous. Which will b e asmall number of include files, and pull the ocrrect one into each page. I'll give that a go.....

Just been discussing tagged malformed permissions with Tariq and what to do with them when dereferencing. Keeping in line with the GCPERM and ACPERM behaviour, if ACPERM could not have created the permissions, all permissions should be treated as 0.
For instance, a tagged cap with ASR + load + store (and no execute) will cause a permissions violation if you try to dereference it.

Yes, sorry, I was imagining that behaviour, but I guess we didn't really define it anywhere. I think the idea of malformed caps is a little confusing because it covers {malformed bounds, reserved permissions encodings, reserved bits set}. It sounds like we're converging on the behaviour that {malformed bounds => read as zero and fail all bounds checks, reserved perms => read as zero and fail all perm checks, reserved bits => don't let the tag become set (with CBLD) but otherwise ignore}


include JavaMoney library fix

see also https://github.com/commercetools/commercetools-sunrise-java/issues/446

build a JVM SDK helper

see also https://github.com/commercetools/commercetools-sunrise-java/pull/441/files

Provided by the JVM SDK already. Additionally, we will investigate further the issue behind that.


[docs] Clarify validation errors and fix link for useDefaults

What issue does this pull request resolve?
I ran into what is essentially https://github.com/ajv-validator/ajv/issues/1790, where I was confused that validation failed but the errors property on the validation function was not set. I have tested this and it appears this is due to a missing, but not required, property in the validated object.
What changes did you make?
Fixes a broken link to the option and adds a clarifying note regarding the errors property in this case.
Is there anything that requires more attention while reviewing?
Ensure I'm correct about this behavior :-)

I don't think the correction is correct, #1790 was related to ajv v6.


Microblocks : Add 2 balances , 1 anchored 1 unanchored

We are now fetching all api endpoint with unanchored=true see https://github.com/blockstack/stacks-wallet-web/pull/1397
So the balance will take into account transactions from microblocks. But this balance is not final (there could be a chance that those microblocks transactions might be reorg).
We decided to show both balances the 'anchored' and 'unanchored' balances

@jasperjansz @aulneau @kyranjamie @fbwoolf
Below a first suggestion

The issue is that the user has no clue which is which. Any suggestion ?
When I look at it, it looks confusing now. Is it still a good idea to show both balances ?
I think showing only the unanchored balance might be a better UX and the user will have the feeling that everything is fast.

@beguene

We decided to show both balances the 'anchored' and 'unanchored' balances

I don't recall this and I don't think it's a good idea for the same reason you mentioned — it's confusing. I think we should have one balance per token and it should update whenever there's a mutation in balance, regardless of whether that's the result of a microblock or anchor block. What do you think?

@jasperjansz I totally agree. That's what I meant in my last sentence.

Let's update this issue title to clarify that we want only one balance (comprising the net results of both microblock-included and anchor-block-included transactions) for each asset?


[Reproduce] Cannot Reproduce the results.

Hello. Thank you for the great work.
However, under the given instructions with the pretrained weight, I could not reproduce the reported result on FSC-147.
I followed your instructiond and my test results are "Current MAE: 23.07, RMSE: 104.73".
My results are so far away from your results in the paper. Is there any hyper-parameters different from yours?

Could you provide further information about your results? For example, which checkpoint did you use and did you fine-tune the provided weights?

Hi, Chang
I download the ckpt from MAE, pretrain and fineturn in FSC with your code.
The environment you provided cannot be installed exactly the same, I
intsall the closest versions.
I test with the 'checkpoint__finetuning_minMAE.pth'
Cheers,
Yuhao
Chang Liu @.***> 于2023年7月12日周三 17:14写道：

Could you provide further information about your results? For example,
which checkpoint did you use and did you fine-tune the provided weights?
—
Reply to this email directly, view it on GitHub
https://github.com/Verg-Avesta/CounTR/issues/32#issuecomment-1632012271,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALAFVAYNAOAJN33VHGTQIRDXPZIVPANCNFSM6AAAAAA2HBULHU
.
You are receiving this because you authored the thread.Message ID:
@.***>

I am not sure what the problem could be now.
Maybe you can try whether the fine-tuned weights I provided could produce the correct results? If the results are close, the environment should be OK.
After that, you can also try to visualize the performance of the model after MAE pre-training, and check whether the model has learned to reconstruct the masked images.

Hi
I use the wanb to record, the visulisation is okay, both reconstruction and
density map.
I dont think package version can lead so large gap.
How many times did you repeat your experiment? According to the issues on
github, the tolerance seems to be large.
Also, i will try the ckpt you provided.
Cheers
Chang Liu @.***> 于2023年7月12日周三 17:31写道：

I am not sure what the problem could be now.
Maybe you can try whether the fine-tuned weights I provided could produce
the correct results? If the results are close, the environment should be OK.
After that, you can also try to visualize the performance of the model
after MAE pre-training, and check whether the model has learned to
reconstruct the masked images.
—
Reply to this email directly, view it on GitHub
https://github.com/Verg-Avesta/CounTR/issues/32#issuecomment-1632034891,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ALAFVA4OZBHIR7EQ3BP6MCTXPZKWTANCNFSM6AAAAAA2HBULHU
.
You are receiving this because you authored the thread.Message ID:
@.***>

I think an MAE difference between -1~1 should be OK. But your result differs too much.....
Besides, I have another suggestion. You can remove this line to unfreeze the encoder, and this will lead to an improvement in performance. But I don't think this will cause such a large performance gap.

Hello,
I am not quite sure what the problem could be. Maybe you can try to remove this line and unfreeze the  image encoder? You can also use a smaller learning rate and check the results of some earlier checkpoints.


Prevent timing side channel attack.

As per #16 not only is '==' lazy and short circuits, but so is all? and any?
so the previous fix won't work for padded attacks (where the string length is
padding during a brute force). This refactoring uses inject which will always
iterate over all the characters.
See https://gist.github.com/JonRowe/6437213
/ht @samphippen for the check

Closed as stale


build(Makefile): more docker env and repo url override

Description
Those are small changes to the main Makefile that would make life easier :

it enables to specify an alternate base URL for the produced images. This is handy when using a local repo.
forward SHORTCOMMIT inside the context of dockerized build. This simplifies building in git worktrees, where git is not available from within the container.

Testing
I expect those changes to be no-op in the usual build environment.

/retest


Exporting adata.X and gene names to csv

Hi all,
Someone else have asked similar questions already but I've been wondering if it's possible to extract data matrix and gene names only into a csv? I've used this to export adata.X into csv (but as expected it was only data matrix without any indication which column shows which gene)
data = adata.X
df = pd.DataFrame(data.toarray())
df.to_csv()
compression_opts = dict(method='zip',
                       archive_name='df.csv')
df.to_csv('df.zip', index=False,
         compression=compression_opts)

I was wondering whether it is possible to extract only gene names from adata.var with adata.X (without the rest of the metadata as it's not important).
Thank you!

If you'd like to write everything (including values in obsm, obs, etc.) to CSVs you can use adata.write_csvs. If you just want X with labels, I would use the to_df() method to get a dataframe of X with obs_names and var_names as the index and columns. E.g.:
adata.to_df().to_csv(...)


Use Fusuma with Objective-C

I saw a couple of issues with using Fusuma in an Objective-C project here and here but they were closed.
Does anyone know how to use Fusuma with Objective-C? Or would they mind writing some instructions?

pod gave me 1.0.3 by default, the issue seems to be fixed in 1.0.4.
I just changed the line in my Podfile to
pod 'Fusuma', :git => 'https://github.com/ytakzk/Fusuma.git'
Now everything is properly exposed to obj-c.

@dsouzarc As @j005u said, it is already fixed 👍