InvalidCastException on Windows Server 2008 R2

If I run my project on a Windows Server 2008 R2 machine I get this Exception:
System.InvalidCastException: Die angegebene Umwandlung ist ungültig.
bei Microsoft.Dism.DismApi.ThrowIfFail(Func`1 func, Int32[] successCodes) in :Zeile 0.
bei Akzentum.InstallHelper.FeatureManipulator.ManipulateWindowsFeature(String[] features)
Feature: IIS-WebServerRole
Package: Microsoft-Windows-Foundation-Package~31bf3856ad364e35~amd64~~6.1.7601.17514
Befor I could run my project I had to install the ADK from Microsoft, because the dismapi.dll isn't installed by default.
https://www.microsoft.com/de-de/download/details.aspx?id=39982
On a Windows Server 2012 and higher everything works fine so far.
Kind regards.

Just to confirm, the error you cited above is what's thrown when you call DismApi.Initialize(), correct?
I believe you're having this problem because DISM isn't natively available on Server 2008 R2, and is instead in a separate Program Files folder.
If you can verify, I'm looking into solution(s).

Hello Shidell,
I'm very sorry that it took me so long to verify your solution, unfortunately I still have this cast issue.
But now I have a bit more detailed Stacktrace, the exceptions happens whild EnableFeature.
System.InvalidCastException: Specified cast is not valid.
bei Microsoft.Dism.DismApi.ThrowIfFail(Func1 func, Int32[] successCodes)    bei Microsoft.Dism.DismApi.ThrowIfFail(Func1 func)
bei Microsoft.Dism.DismApi.EnableFeature(DismSession session, String featureName, String identifier, DismPackageIdentifier packageIdentifier, Boolean limitAccess, Boolean enableAll, List1 sourcePaths, DismProgressCallback progressCallback, Object userData)    bei Microsoft.Dism.DismApi.EnableFeatureByPackageName(DismSession session, String featureName, String packageName, Boolean limitAccess, Boolean enableAll, List1 sourcePaths, DismProgressCallback progressCallback, Object userData)
bei Microsoft.Dism.DismApi.EnableFeatureByPackageName(DismSession session, String featureName, String packageName, Boolean limitAccess, Boolean enableAll, List`1 sourcePaths, DismProgressCallback progressCallback)
bei Microsoft.Dism.DismApi.EnableFeatureByPackageName(DismSession session, String featureName, String packageName, Boolean limitAccess, Boolean enableAll)
bei Akzentum.InstallHelper.FeatureManipulator.ManipulateWindowsFeature(String[] features)
Kind Regards

@Akzentianer
I've created a new branch "add-dism-utilities-support" from my fork. @josemesona and I have been discussing how to properly implement this new "utility" class, to add support for external DISM libraries, in a manner that makes sense:
https://github.com/Shidell/ManagedDism/tree/add-dism-utilities-support
I would invite you to clone this fork and try using the DismUtilities class (read the comments first) to load DISM and report back your findings, as I expect this should solve the problem you're having. but having your experience and testing on 2008 would be a great test.

I still get this error on a Windows Server 2008 R2.
I tracked down where this is happening and I found this line:
return Marshal.GetExceptionForHR(errorCode);
it is in the file DismException.cs line 66.
The issue is that the errorCode is -2147467262.
This is because in this Line:
int hresult = func();
the func return is  -2147467262.
This is in DismApi.cs line 1412.
The reason why the hresult is negativ is because hresult is int and not long , I changed this and everything around the function ThrowIfFail and in the NativeMethods.cs to long (return type only) and the I got displayed this error number "2147500034".
As far as I can see this error is returned from WUSA and means that something is not support.
Kind regards

I believe the InvalidCastException is happening somewhere else.
Although it does appear that the Int32 is overflowing, its still properly represented as -2147467262.  Under the debugger, I can see that these values are equivilent:
int i = -2147467262;
uint x =<PHONE_NUMBER>;

The only difference is how they are printed but they are byte for byte the same

And the Int64 version is also the same byte-wise:
long y =<PHONE_NUMBER>;

All other Dism interop implementations I can find use an Int32 including the PowerShell CmdLets that Microsoft ships.
C:\Windows\System32\WindowsPowerShell\v1.0\Modules\Dism\Microsoft.Dism.PowerShell.dll
// Microsoft.Dism.Commands.DismInterop
using System.Runtime.InteropServices;

[DllImport("dismapi.dll")]
public static extern int DismInitialize(LogLevel LogLevel, [MarshalAs(UnmanagedType.LPWStr)] string LogFilePath, [MarshalAs(UnmanagedType.LPWStr)] string ScratchDirectory);

I haven't been able to repro the issue yet, do you get an error using the PowerShell CmdLet or dism.exe?


Architecture

Hi,
I am trying to re-implement this myself and found your repo.
Could you briefly outline the architecture you followed for this?

Could you elaborate on what architecture means?


Allow BNB Plugin to be Loaded Without PEFT Wrapping

This issue regards to the warnings that for QLoRA PeFT we should pass peft_config directly to SFTTrainer

if no_peft_model: True then model_loader will simply load the BNB model, and some logic needed to set requires_agumentation: False
peft wrapping will be delegated to trf.SFTTrainer
add one test for this new flag in tests/test_peft_plugin
in generate sample configurations update the CONFIGURATIONS and COMBINATIONS:  CONFIGURATIONS = {
    KEY_AUTO_GPTQ: "plugins/accelerated-peft/configs/autogptq.yaml",
    KEY_BNB_NF4: (
        "plugins/accelerated-peft/configs/bnb.yaml",
        [("peft.quantization.bitsandbytes.quant_type", "nf4")],
    ),
   KEY_BNB_NF4_BASELINE: (
        "plugins/accelerated-peft/configs/bnb.yaml",
        [
            ("peft.quantization.bitsandbytes.quant_type", "nf4"),
           ("peft.quantization.bitsandbytes.no_peft_model", True),
         ],
    ),
}

COMBINATIONS = [
    ("accelerated-peft-autogptq", (KEY_AUTO_GPTQ,)),
    ("accelerated-peft-bnb-nf4", (KEY_BNB_NF4,)),
    ("baseline-bnb-nf4", (KEY_BNB_NF4_BASELINE,)),
]

regenerate the benches, update the CSV and  the README

configs/bnb.yaml
add a new flag no_peft_model

# PEFT-related acceleration
peft:

  # quantization-releated acceleration
  # e.g., kernels for quantized base weights
  quantization:

    # For loading BitsAndBytes quantized layers
    # to serve as 4bit base-weights for LoRA PEFT-tuning.
    # NOTE: currently AutoGPTQ is not properly integrated into huggingface /
    # bitsandbytes, thus recommended quant_type to be either "nf4"
    # or "fp4".
    # bitsandbytes:
    bitsandbytes:
      quant_type: nf4

      # If True, then no get_peft_model and prepare_model_for_kbit_training
      # will be called.
      no_peft_model: False

@achew010 note that we put in a comment in the main README

Huggingface BNB QLoRA numbers taken with legacy approaches, but we are aware of https://github.com/foundation-model-stack/fms-acceleration/issues/10 and will update our benches. The above includes numbers using fusedOps-and-kernels and actual impl coming soon, see below.

Now that this issue is closed, that comment should be removed.


Unexplained 404 on Index.html

Hey team,
We have encountered a 404 error when loading the home screen (aka the / path) of the Django site. Currently the cause of this error is unknown. I will paste the error message from the Chrome Inspect tool's console below, please help us out if you know what this means.
Error Message
base.js:1550 GET https://r5---sn-p5qs7nek.googlevideo.com/videoplayback?expire=1603313993&ei=6UyQX-vtLOrr8gTm-46gAg&ip=<IP_ADDRESS>&id=o-AAbIqadU2RuqH9IKsTCbYMIUA8wlzIRuvwi_Gu4UPFap&itag=135&aitags=133%2C134%2C135%2C136%2C160%2C242%2C243%2C244%2C247%2C278%2C298%2C302&source=yt_otf&requiressl=yes&mh=SM&mm=31%2C29&mn=sn-uvfopn2-cvne%2Csn-p5qs7nek&ms=au%2Crdu&mv=m&mvi=2&pcm2cms=yes&pl=24&initcwndbps=1701250&vprv=1&mime=video%2Fmp4&otf=1&otfp=1&dur=0.000&lmt=1578890863547100&mt=1603292336&fvip=5&keepalive=yes&fexp=23915654&c=WEB_EMBEDDED_PLAYER&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cotf%2Cotfp%2Cdur%2Clmt&sig=AOq0QJ8wRQIgeOeeFNgeX0OwXEcn31qErFYuSJl7RCxCg3Eb14WJsiwCIQCWNKovisqIVzNx41lPQ3gwJ44fucZRvM303fX8HCO0vA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAKN072XnWXQRky_5by2bD6iTFDhB14YyccwdCwfgVzQLAiEA03L4ygFKEggt8dsaUVZjgmgw-K2YXsC_WC5iGllwOnw%3D&alr=yes&cpn=PiFr2MYMCnbRlhAp&cver=20201020&fallback_count=1&rn=34&rbuf=0 404

Other details
The error currently isn't causing anything on the UI to be messed up, so my hunch is this error has to do with one of the 3rd-party APIs we have integrated into any of the following HTML files:

carbon0/templates/base.html --> base HTML template
carbon0/templates/index.html --> main HTML template used for this view. It integrates the Mixpanel Javascript SDK at the top.
carbon0/templates/partials/intro_video_player.html --> partial template that integrates the YouTube iFrame Player API.

Any help is much appreciated!

I think was solved a while ago and I forgot to update the issue. Closing for now, 99% sure it was just some small issue.


error TS2430: Interface 'ConfirmProps' incorrectly extends interface 'ModalProps'.

Steps
Import stuff from a file in project:
import { Grid } from 'semantic-ui-react';

Compile with Typescript
Expected Result
It compiles
Actual Result
It fails to compile. With error:
ERROR in /**REDACTED**/node_modules/semantic-ui-react/dist/commonjs/addons/Confirm/Confirm.d.ts
(4,18): error TS2430: Interface 'ConfirmProps' incorrectly extends interface 'ModalProps'.
  Property 'content' is optional in type 'ConfirmProps' but required in type 'ModalProps'.

Version
0.68.1 has bug, 0.68.0 does not
Testcase
See above

There's a PR open for this, #1611.  Thanks for the report.

Released in<EMAIL_ADDRESS>


GH-43229: [Java] Update Maven project info

Rationale for this change
Some Maven modules are missing project information like the website url, the mailing lists, and scm and issues url
Other may have incorrect links because of the way Maven interpolates those values at build time
What changes are included in this PR?
Update/Fix Maven project information for all modules:

Add project url, mailing lists, scm and issueManagement information to bom and maven parent modules
Fix top-level parent by preventing Maven to rewrite project url, and scm connections/urls based on the module hierarchy
Change project.scm.tag to main and update version change script to also change the tag value to apache-arrow-${version}

Are these changes tested?
CI/CD only
Are there any user-facing changes?
No

GitHub Issue: #43229

It seems the tests for the version bump fail now, though

Yes, a test I didn't know about is failing (which is good, I was worried about the release scripts to not be tested). I will go and fix it

@github-actions crossbow submit -g java

CC @kou for the release scripts

@github-actions crossbow submit -g java-jars

@github-actions crossbow submit -g java


Typo: README Markdown flag typo

Description
There is a typo in the description of the Markdown flag command.

The last command provided uses a --md flag, which is not a valid flag.

Thank you for pointing this out, this was patched in PR #9


HBASE-27423 Upgrade Jackson for CVE-2022-42003/42004

Jackson 2.13.4 fixes CVE-2022-42004 and databind 2.14.0-rc1 fixes CVE-2022-42003.
Move jackson.version to 2.13.4.
Move jackson.databind.version to 2.14.0-rc1.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m  1s
Docker mode activated.

_ Prechecks _

+1 :green_heart:
dupname
0m  0s
No case conflicting files found.

+1 :green_heart:
@author
0m  0s
The patch does not contain any @author tags.

_ master Compile Tests _

+1 :green_heart:
mvninstall
2m 19s
master passed

+1 :green_heart:
compile
6m 21s
master passed

+1 :green_heart:
spotless
0m 40s
branch has no errors when running spotless:check.

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
2m  7s
the patch passed

+1 :green_heart:
compile
6m 26s
the patch passed

-0 :warning:
javac
6m 26s
root generated 2 new + 701 unchanged - 2 fixed = 703 total (was 703)

+1 :green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

+1 :green_heart:
xml
0m  0s
The patch has no ill-formed XML file.

+1 :green_heart:
hadoopcheck
7m 59s
Patch does not cause any errors with Hadoop 3.2.4 3.3.4.

+1 :green_heart:
spotless
0m 39s
patch has no errors when running spotless:check.

_ Other Tests _

+1 :green_heart:
asflicense
0m 13s
The patch does not generate ASF License warnings.

32m 59s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4821/1/artifact/yetus-general-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/4821

Optional Tests
dupname asflicense javac hadoopcheck spotless xml compile

uname
Linux eefa38f0e25f 5.4.0-124-generic #140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
master / 8d2efc8aaa

Default Java
Temurin-1.8.0_345-b01

javac
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4821/1/artifact/yetus-general-check/output/diff-compile-javac-root.txt

Max. process+thread count
139 (vs. ulimit of 30000)

modules
C: . U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4821/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

2.14.0-rc2 is out, maybe we should wait for 2.14.0 final release and then upgrading?  U: .

 U: .

Console output
https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4821/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

2.14.0-rc2 is out, maybe we should wait for 2.14.0 final release and then upgrading?

Sure we can wait. I'll move this to draft and reschedule the JIRA.

Will need a new thirdparty release then...

You mean because of hbase-shaded-jackson-jaxrs-json-provider I presume. Ok.

Fixed by #4878


How do you hide and show the modal via Javascript?

How do you hide and show the modal via Javascript?
let myModal
if (desc === 'short'){
   myModal = document.getElementById('shortDesc')
} else {
  myModal = document.getElementById('desc')
}
// eslint-disable-next-line no-undef
const modal = bootstrap.Modal.getInstance(myModal)
modal.show()

Using the above I get an error:
Uncaught (in promise) ReferenceError: bootstrap is not defined

Hey @mudenho
The error is given because you have not imported the bootstrap. Try importing the bootstrap in your script and let me know how it goes.
Regards,
Maisam Afshar

Hey @mudenho
The error is given because you have not imported the bootstrap. Try importing the bootstrap in your script and let me know how it goes.
Regards, Maisam Afshar

Perfect. That worked. Thank you.


Attempt to strip armv7 artifacts

Manual run: https://fusion2.corp.google.com/invocations/81e1ad8a-35c0-4537-96e4-8f5c02bc7c8f/targets

Looks like we managed to shrink the artifacts by about 25x.
This is the other half of the long-term solution for the problem mitigated by https://github.com/pypi/support/issues/3616


How to use by python

Brother can u plz make a video on how to use it by python.  I cant use docker as my system is on. Windows 7.  I have tried ur instructions but not able to undersrand

docker does not work in windows 7.
follow below link to install python on windows. it will also install pip.
python on windows
Once the installation is complete, you can either clone this repository to cowin-app(git is required.) OR you can download this repository in zip format.
If you download the zip format then unzip it and change folder name to cowin-app. After this step, open command line interface  and then cd to the cowin-app folder. Your current working directory should be cowin-app folder. and then run pip install -r requirements.txt. This will install all prerequisites.
Now cd to that folder where cowin-app folder is present and not inside the cowin-app folder. And run bokeh serve --port 5100 cowin-app/ for manual mode and for auto mode run bokeh serve --port 5100 cowin-app/ --args<EMAIL_ADDRESS>"app_password_here"

docker does not work in windows 7.
follow below link to install python on windows. it will also install pip.
python on windows
Once the installation is complete, you can either clone this repository to cowin-app(git is required.) OR you can download this repository in zip format.
If you download the zip format then unzip it and change folder name to cowin-app. After this step, open command line interface and then cd to the cowin-app folder. Your current working directory should be cowin-app folder. and then run pip install -r requirements.txt. This will install all prerequisites.
Now cd to that folder where cowin-app folder is present and not inside the cowin-app folder. And run bokeh serve --port 5100 cowin-app/ for manual mode and for auto mode run bokeh serve --port 5100 cowin-app/ --args<EMAIL_ADDRESS>"app_password_here"
for auto mode please go through documentation

Thank u so much brother for explaining in this manner. I will tell us surely after installing the same using this method. Thanks again

@suraj-deshmukh thanks buddy i was able to run repo following ur instructions. I have 1 question if we are booking multiple beneficiaries then whether we have to write name of all the beneficiaries in name column or just 1 name and type beneficiaries id in its colums seperated by comma.

@suraj-deshmukh thanks buddy i was able to run repo following ur instructions. I have 1 question if we are booking multiple beneficiaries then whether we have to write name of all the beneficiaries in name column or just 1 name and type beneficiaries id in its colums seperated by comma.

name  is just placeholder, so just put your name or anything. It does not matter at all what you put there. In case of multiple beneficiaries, use csv format like<PHONE_NUMBER>1234,<PHONE_NUMBER>1234. for multple csvs use the same format 411002,411003 etc.
Note: beneficiary id is 14 digit long number
Note: select correct district where those pincodes belong. When you select state and district. Background thread pulls data from that district and filters the data using given pincodes.

@suraj-deshmukh thanks buddy. I have starred ur repo.  Thanks again. I hope i vill b able to book slot using ur repo

@520rahul yes it works provided you have given all the input correct way. If you have selected manual way then stay alert to put otp in dashboard once you get otp on mobile and then click Submit otp and book button.


Edit typo for AML compute section on strict limits

@brendalee : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@blackmist - hi! can you help approve - fixed a small typo. thank you!

Looks good, #sign-off

@Blackmist: I'm sorry - only the author of this article, @nishankgu, can sign off on your changes. But we do have an exception process - if you are on the Microsoft content or product team for this product area, you can ask the PR review team to review and merge it by sending mail to the techdocprs alias.

Thanks for helping improve the docs @brendalee !


Use gunicorn as the web server

Flask recommends that you do not use their built-in server in production: https://flask.palletsprojects.com/en/master/deploying/#deployment-options. I went with the first standalone option listed on their page: gunicorn.

If this means we won't be able to recognize the user from the logs, I think we should fix that first.

Looks like the logs lack the GET parameter.
If this means we won't be able to recognize the user from the logs, I think we should fix that first.

Fixed with --access-logfile
<IP_ADDRESS> - - [13/Aug/2020:17:32:43 +0000] "GET /dataset/dataset HTTP/1.1" 302 357 "http://localhost:5000/datasets?user=21cb59e575cd4f83bc98cac289b7ec7a" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"
[2020-08-13 17:32:43 +0000] [10] [DEBUG] GET /dataset/dataset
<IP_ADDRESS> - - [13/Aug/2020:17:32:43 +0000] "GET /dataset/dataset?user=21cb59e575cd4f83bc98cac289b7ec7a HTTP/1.1" 200 3571 "http://localhost:5000/datasets?user=21cb59e575cd4f83bc98cac289b7ec7a" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.105 Safari/537.36"

I also verified that Flask exceptions show up in the logs


s3gateway: support for versioning

Closes #3902
Currently diffed against s3gateway-multipart to be readable. Blocked by #3906.

I'm not sure if this is the right path forward due to the following reasons:

As @jdoliner mentioned, this means that working with non-master branches will be not possible for most operations when using a library.
s3 includes the notion of delete markers, which PFS does not have.
In order to be s3 complaint, we'd need to make heavy use of history and commit inspection, which would carry substantial performance implications.
This would change buckets to map to repos rather than branches. This has some nice ergonomic/aesthetic aspects to it, but would be a major backwards-incompatible change.

Still mulling on ways to make this less of a square peg / round hole issue.

I'm working on an alternative path, where we maintain the branch.repo bucket naming convention. This will solve issues (1) and (4).


fix: error: Invalid template interpolation value

Description
Issue: https://github.com/terraform-ibm-modules/terraform-ibm-icd-elasticsearch/issues/296

Release required?

[ ] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

/run pipeline

Strange, the fscloud example failed on idempotent failure:
        	Messages:   	Resource(s) identified to be updated
        	            	Name: elasticsearch_tag
        	            	Address: module.elasticsearch.module.elasticsearch.ibm_resource_tag.elasticsearch_tag[0]
        	            	Actions: [update]
        	            	DIFF:
        	            	  Before:
        	            		{"tags":"SECURE_VALUE_HIDDEN_HASH:-f21d72e82ece0452202370f3b19ecf8dcab590f0d221203bd288dc7e"}
        	            	  After:
        	            		{"tags":"SECURE_VALUE_HIDDEN_HASH:-48b3f952b63bfa385b736ec6d98bde68b37ed7f3528ea11bea11b9ed"}

Going to retry..

/run pipeline

│     "StatusCode": 410,
│     "Headers": {
│         "Cache-Control": [
│             "max-age=0, no-cache, no-store"
│         ],
│         "Content-Length": [
│             "3"
│         ],
│         "Content-Type": [
│             "application/json"
│         ],
│         "Date": [
│             "Wed, 25 Sep 2024 13:30:43 GMT"
│         ],
│         "Expires": [
│             "Wed, 25 Sep 2024 13:30:43 GMT"
│         ],
│         "Pragma": [
│             "no-cache"
│         ],
│         "Request-Id": [
│             "626c-8a855dccbd3ba318"
│         ],
│         "Retry-After": [
│             "0"
│         ],
│         "Server": [
│             "istio-envoy"
│         ],
│         "Set-Cookie": [
│             "ak_bmsc=082C2EFFB8580EB188A11BCA9572D579~000000000000000000000000000000~YAAQk2rcF7dihyeSAQAA5IBgKRm+44slqhnwjI+cmziOtZ8h64ilOY2m2M7hn5L75hqCchTn//AmJzGQYij7kEtXzD5/mQ84xs9uAdJ6slkzacVzvbukIyXONKREm0vkDeGQnGggJMZar6bnHPWHK77TP8ze6MDKyg0RNvlOiRkR+lNSnWiqePR5CE4FnTsIiPA2AU/WoVE0K0jydnHvl4Q4/5WEY3gZIdU58bhxo2O+eV+lpyV1Dexp+Gek0wISfyrUUhl6YOktpccjkBvxpkLKEJcwDXmM7I/cFRfX+oryZJ9AWbL7B08/0Ni1ASt5G0Kx9EhxbJCuYq5NoYjOnhojdN7PUIbrEvEZ0ngAjsgkySq/iXoS0DPaScqAUYmUzhBHbd7ehXGt0y/U; Domain=.cloud.ibm.com; Path=/; Expires=Wed, 25 Sep 2024 15:30:43 GMT; Max-Age=7200"
│         ],
│         "Strict-Transport-Security": [
│             "max-age=31536000;includeSubDomains"
│         ],
│         "Transaction-Id": [
│             "626c-0adf0ac086a5044f"
│         ],
│         "X-Content-Type-Options": [
│             "nosniff"
│         ],
│         "X-Correlation-Id": [
│             "626c-0adf0ac086a5044f"
│         ],
│         "X-Envoy-Upstream-Service-Time": [
│             "76"
│         ],
│         "X-Op-Completion-Time": [
│             ""
│         ],
│         "X-Ratelimit-Limit": [
│             "100"
│         ],
│         "X-Ratelimit-Remaining": [
│             "95"
│         ],
│         "X-Ratelimit-Reset": [
│             "0"
│         ]
│     },
│     "Result": {},
│     "RawResult": null
│ }
│
│

retrying..

/run pipeline

:tada: This PR is included in version 1.20.1 :tada:
The release is available on:

GitHub release
v1.20.1

Your semantic-release bot :package::rocket:


feat: iterator over rust range

#42

Memory usage:

red - openpyxl, read_only=True, iter_rows()
black - master, to_python()
blue - this PR, iter_rows()

example file (saved as xlsx in LibreOffice).
TODO:

Fix empty cells and tests

Any updates?

Thanks for putting together the PoC. This would be really cool to see available in the official package.
So.... I'm bumping this. Any updates?


Payload size is limited in 31bytes (248bit), witch is not enough for Format 3 subformat 1 - Trap

The max size of Format 3 subformat 1 - Trap is 330 bit theoretically.
<Unformatted Trace & Diagnostic Data Packet Encapsulation for RISC-V> limit the payload size in 248 bit. Do we need to send two packets separately?

I'd be interested to see how you got to 330 bits.  But yes, if the payload exceeds 248 bits it would need to be split into 2.  There are various ways this could be accommodated, but I don't think it should be necessary as I don't believe any real-world system is going to get anywhere near the 248-bit limit.
Iain
From: zhangdujiao @.>
Sent: 05 June 2024 10:08
To: riscv-non-isa/riscv-trace-spec @.>
Cc: Subscribed @.***>
Subject: Re: [riscv-non-isa/riscv-trace-spec] Payload size is limited in 31bytes (248bit), witch is not enough for Format 3 subformat 1 - Trap (Issue #114)
The max size of Format 3 subformat 1 - Trap is 330 bit theoretically.
<Unformatted Trace & Diagnostic Data Packet Encapsulation for RISC-V> limit the payload size in 248 bit. Do we need to send two packets separately?

Reply to this email directly, view it on GitHubhttps://github.com/riscv-non-isa/riscv-trace-spec/issues/114#issuecomment-2149279655, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQOPSUTXTKU2IRO6OC4K4DZF3IPDAVCNFSM6AAAAABI2INYJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBZGI3TSNRVGU.
You are receiving this because you are subscribed to this thread.Message ID<EMAIL_ADDRESS>

I'd be interested to see how you got to 330 bits.

For example:
Format 3 subformat 1 - Trap
"
format 2bit
subformat 2bit
branch 1bit
privilege 3bit
time 64 bit
context 64 bit
ecause 64 bit
interrupt 1 bit
thaddr 1bit
address 64bit
tval 64bit
"
The payload size 330bits total.
We can put timestamp in ATDATA rather than payload, while the payload is 266 bits then, still larger than 248 bits.

I have never seen any suggestion that ecause is more than 6 bits wide. Currently any codes > 64 are reserved, and even if this changes at some point the total number of bits might increase to 7 or 8, but not 64 (actually 63, as the MSB of the xcause registers is used for the interrupt bit).

Got it, we need to re-considerate the width of port signals, many thanks Iain.

the MSB of the xcause registers is used for the interrupt bit

The ingress port signal cause includes the value from ucause/scause/ vscause/mcause, and the MSB of the xcause registers is used for the interrupt bit. So, the whole xcause value should be included in the packet. Otherwise, the decoder cannot determine the cause as an interrupt or an exception, right?

Yes.  But you only need as many bits in ecause as there are defined exception causes.  This is currently a max of 6 bits, and in some implementations only 5.
Iain
From: zhangdujiao @.>
Sent: 07 June 2024 04:33
To: riscv-non-isa/riscv-trace-spec @.>
Cc: Robertson, Iain (DI SW ICS TST RD EAH) @.>; Comment @.>
Subject: Re: [riscv-non-isa/riscv-trace-spec] Payload size is limited in 31bytes (248bit), witch is not enough for Format 3 subformat 1 - Trap (Issue #114)
the MSB of the xcause registers is used for the interrupt bit
The ingress port signal cause includes the value from ucause/scause/ vscause/mcause, and the MSB of the xcause registers is used for the interrupt bit. So, the whole xcause value should be included in the packet. Otherwise, the decoder cannot determine the cause as an interrupt or an exception, right?

Reply to this email directly, view it on GitHubhttps://github.com/riscv-non-isa/riscv-trace-spec/issues/114#issuecomment-2153784299, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQOPSTW7VGPMOTSAUAKAPLZGESXDAVCNFSM6AAAAABI2INYJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJTG44DIMRZHE.
You are receiving this because you commented.Message ID<EMAIL_ADDRESS>

But you only need as many bits in ecause as there are defined exception causes. This is currently a max of 6 bits, and in some implementations only 5.

If only the LSB 6 bits of xcause are contained in packet, instead of all 64 bits, can we determine whether the xcause value indicates an interrupt or an exception cause? Can the interrupt field in Trap packet indicate an interrupt or exception?

Yes, because the MSB is connected to the 'interrupt' signal in the interface.
Iain
From: zhangdujiao @.>
Sent: 07 June 2024 09:01
To: riscv-non-isa/riscv-trace-spec @.>
Cc: Robertson, Iain (DI SW ICS TST RD EAH) @.>; Comment @.>
Subject: Re: [riscv-non-isa/riscv-trace-spec] Payload size is limited in 31bytes (248bit), witch is not enough for Format 3 subformat 1 - Trap (Issue #114)
But you only need as many bits in ecause as there are defined exception causes. This is currently a max of 6 bits, and in some implementations only 5.
If only the LSB 6 bits of xcause are contained in packet, instead of all 64 bits, can we determine whether the xcause value indicates an interrupt or an exception cause? Can the interrupt field in Trap packet indicate an interrupt or exception?

Reply to this email directly, view it on GitHubhttps://github.com/riscv-non-isa/riscv-trace-spec/issues/114#issuecomment-2154304210, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALQOPSUGOTSR7ZUEZEJOQQDZGFSDVAVCNFSM6AAAAABI2INYJSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJUGMYDIMRRGA.
You are receiving this because you commented.Message ID<EMAIL_ADDRESS>

Yes, because the MSB is connected to the 'interrupt' signal in the interface.

Got it! Many thanks!


Include Kotlin DSL samples for Ant chapter

Context
#6442 Include Kotlin DSL samples in all user manual chapters
Contributor Checklist

[X] Review Contribution Guidelines
[X] Make sure that all commits are signed off to indicate that you agree to the terms of Developer Certificate of Origin.
[X] Check "Allow edit from maintainers" option in pull request so that additional changes can be pushed by Gradle team
[X] Ensure that tests pass locally: ./gradlew :integTest:integTest --tests "UserGuideSamplesIntegrationTest"

Gradle Core Team Checklist

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation
[ ] Recognize contributor in release notes

Thank you @jjustinic! This was squashed and merged with 7f3440d after a passing branch build


Rename cfg(loom_crossbeam) to cfg(crossbeam_loom)

It matches with the convention that will be documented in loom. See https://github.com/crossbeam-rs/crossbeam/pull/487#issuecomment-753800120 for more.
r? @jeehoonkang @jonhoo

bors r=jonhoo


ONNX tests: exclude all unsupported filetype tests

There are ONXX tests for regular ops with a unsupported data types, such as test_max_uint8_cpu (int16, unint8, uint16, uint64, float64, ...). They don't get excluded by the current regex expressions.
I changed the exclusion regex, to include all tests containing these datatypes in their name, although I could write out all the tests in the format of test_<OP>_<TYPE>*, for clarity as well.
PASSED count is same pre & after the change (doesn't exclude any passing tests), but SKIPPED goes 1862 -> 1885.
Is a small change, but the first thing I noticed for getting to 0 FAILED. If some of these data types should work, I could look into making the tests PASS too.

Yea, I think getting to 0 failed is a good goal if things are obviously unsupported.


Suppress by error code

Extracted from https://github.com/microsoft/vscode/pull/108682:

@jrieken: Would it be possible to teach tsec more than one error code and to able to suppress certain error codes (instead of files)?

The use case Johannes outlined in vscode is their usage of dynamic imports - there's nothing actionable for the projects to move off them, but tsec should be able to surface violations of a different kind in the files.

Maybe we can support wildcards in the exemption list, so that we can have something like
{
    "ban-dynamic-imports": ["**"]
}


Add airflow-version + chart-version support matrix

Currently, users are confused how they can run versions of airflow/python which are not the default ones specified by the airflow.image.* values, we should add this info to the readme.
We should also add a support matrix showing "airflow" and "chart" version combinations which are supported and not-supported. (And explicitly state the limitations of our 1.10.X support)

This is still an issue, reopening.


Using discrete Graphics

Messages has started to switch to using the discrete graphics card when opening. What caused it?

There seems to be 4 instances of Messages Helper going everytime I open the app. What is happening here?


Update references to /etc/mobile

(fixes #138)

Codecov Report

Merging #139 into development will not change coverage.
The diff coverage is n/a.

@@              Coverage Diff               @@
##             development     #139   +/-   ##
==============================================
  Coverage          82.49%   82.49%
  Complexity           736      736
==============================================
  Files                 71       71
  Lines               2393     2393
  Branches             427      427
==============================================
  Hits                1974     1974
  Misses               170      170
  Partials             249      249

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f9268c8...4f9309d. Read the comment docs.

does that work on AEM 6.3? w

@keepthebyte, I verified with 6.3 SP1 and the emulators are being picked correctly from /etc/mobile/groups/responsive with this change.

@keepthebyte, you are actually right: relative paths are not supported in 6.3, they will be supported starting with 6.4.
However, given this is test content and our tests do not check responsive functionality (out of scope for the components) and pass with these changes, we would prefer to update the paths to the cleaner version, just in case new projects targeting 6.4 get inspiration from our content.

@vladbailescu If this really works only with AEM 6.4 at least a short comment should be placed in that test content.

The content installs in 6.3 and produces no errors. However, using a relative path for for the responsive mobile groups will cause the mobile emulators to not load. Testing the emulators is not in scope of the core components so arguably the change works for this project in 6.3 too.


Update mockito-core to 3.6.0

Updates org.mockito:mockito-core from 3.5.15 to 3.6.0.
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.mockito", artifactId = "mockito-core" } ]

labels: test-library-update, semver-minor

We string this thing along for one spec.  It would be neat to eliminate it.


Camera not starts on web on ios

Camera not starts (full white screen) when testing on Safari on iOS for web

Could you provide the logs from the browser console?
You can access these by

connecting your iOS device to a Mac
open Safari
select the Develop option in the MacOS toolbar
open the JS console for the attached device

Apparently this is a duplicate of https://github.com/juliansteenbakker/mobile_scanner/issues/950
Going to close this issue, so that we can follow up there.


Enable once attribute struct is supported

https://github.com/project-chip/connectedhomeip/blob/bebbd74187a721154f82061f1023a347d7ce7f57/src/app/tests/suites/certification/Test_TC_MC_9_1.yaml#L68-L73

This issue was generated by todo-issue based on a TODO comment in bebbd74187a721154f82061f1023a347d7ce7f57.

Struct attributes are supported, but reading this attribute from all-clusters-app always returns failure.  This test should probably run against tv-app?


Plugable coins API

Finalization todo list:

[x] Revert Wallet GATEWAY validation and usage (@mnaichuk)
[x] Remove BlockchainAddress, use Hash instead (@mnaichuk)
[x] Finish WalletService2 specs (@mnaichuk)
[x] https://github.com/rubykube/peatio/pull/2168/files#r275769063 (@mnaichuk)
[x] Build Exception hierarchy for Blockchain & Wallet
[x] Add Peato::Blockchain::Abstact #load_balance

[x] Add to Bitcoin
[x] Add to Ethereum (@mnaichuk)

[x] Replace Blockchain #blockchain_api with v2
[x] Remove all currencies except btc eth & erc20 from specs
[x] Implement Ethereum::Wallet adapter
[x] Integrate substract_fee option into bitcoin with specs(@mnaichuk)
[x] Update daemons to use WalletService2 & BlockchainService2

[x] Blockchain daemon with specs(@mnaichuk)
[x] Withdraw daemons with specs

[x] Finalize TODOs

[x] Remove unused resources/
[x] Finish TODOs in specs
[x] Uncomment wallet validation

[x] Move core code to gem
[x] Functional testing in microkube

[x] Without plugins
[x] With plugin
[x] Final fixes

[x] Finalize

[x] Release peatio 0.5.2 and
[x] Update peatio to 0.5.2 in peatio-litecoin
[x] Release peatio-litecoin
[x] Upeate peatio & peatio-litecoin gems in pr

closed #2045
closes #2155

TODO:

[ ] blockchain_service2_spec.rb is failing in case of --seed 33495

[ ]  Deal with redis cleaning

[ ]  TODO: Remove me once we replace Blockchain#blokchain_api with Blockchain#blockchain_apiv2.

[ ] Finalize Abstract documentation

Finalization todo list:

[x] Revert Wallet GATEWAY validation and usage (@mnaichuk)
[x] Remove BlockchainAddress, use Hash instead (@mnaichuk)
[x] Finish WalletService2 specs (@mnaichuk)
[x] https://github.com/rubykube/peatio/pull/2168/files#r275769063 (@mnaichuk)
[x] Build Exception hierarchy for Blockchain & Wallet
[x] Add Peato::Blockchain::Abstact #load_balance

[x] Add to Bitcoin
[x] Add to Ethereum (@mnaichuk)

[x] Replace Blockchain #blockchain_api with v2
[x] Remove all currencies except btc eth & erc20 from specs
[x] Implement Ethereum::Wallet adapter
[x] Integrate substract_fee option into bitcoin with specs(@mnaichuk)
[x] Update daemons to use WalletService2 & BlockchainService2

[x] Blockchain daemon with specs(@mnaichuk)
[x] Withdraw daemons with specs

[x] Finalize TODOs

[x] Remove unused resources/
[x] Finish TODOs in specs
[x] Uncomment wallet validation

[x] Move core code to gem
[x] Functional testing in microkube

[x] Without plugins
[x] With plugin
[x] Final fixes

[x] Finalize

[x] Release peatio 0.5.2 and
[x] Update peatio to 0.5.2 in peatio-litecoin
[x] Release peatio-litecoin
[x] Upeate peatio & peatio-litecoin gems in pr

closed #2045
closes #2155


Update the SQLite dependency to support Android 7

/cc @ghuntley @ericsink Can you review this please?

Re:  The failure of the automatic checks:  It looks to be a problem with a missing Android API in the SDK on the build machine.  If that's related to SQLitePCL.raw, I do not yet see how.
Re:  The PR itself:
If you have a specific preference for using the system SQLite [only] on iOS, then it looks fine.  Otherwise, I suggest using bundle_e_sqlite3 instead of bundle_green.
Did the unit tests pass for you with this change?  Just curious -- @ghuntley had a problem with that when he attempted something similar.

Re: The failure of the automatic checks: It looks to be a problem with a missing Android API in the SDK on the build machine. If that's related to SQLitePCL.raw, I do not yet see how.

@ghuntley Could you take a look at that?

If you have a specific preference for using the system SQLite [only] on iOS, then it looks fine. Otherwise, I suggest using bundle_e_sqlite3 instead of bundle_green.

Alright, I can do that, I just had a look Azure Mobile SDK at what they are doing, and it seems like they're using green

Did the unit tests pass for you with this change? Just curious -- @ghuntley had a problem with that when he attempted something similar.

Yup, they're working now, I think @ghuntley just forgot to update the dependencies of the test project, so it still used the old raw version

If you have a specific preference for using the system SQLite [only] on iOS, then it looks fine. Otherwise, I suggest using bundle_e_sqlite3 instead of bundle_green.
Alright, I can do that, I just had a look Azure Mobile SDK at what they are doing, and it seems like they're using green

Ugh damn; I was wanting to use bundle_e_sqlite3 on iOS for consistency in bug reports across all platforms but looks like Azure Mobile SDK has tied our hands in this matter?

I dislike the idea that all SQLitePCL.raw-using libs have to choose the same bundle.  That should not be necessary.
And if it were necessary, I would dislike it even more if bundle_green were the one everyone chose.
Use bundle_e_sqlite3.
But, to answer your question:
If two libs are using two different bundles, the last call to Batteries.Init() wins.  In the example you gave, this could mean that Akavache might end up using the iOS system SQLite, or that Azure Mobile SDK might end up using e_sqlite3, neither of which would be a tragedy.
If either "last call wins" or "first call wins" were clearly more correct, I would choose it, but this choice seems arbitrary.

Switched to bundle_e_sqlite3 and the unit tests are passing. Thank-you Eric for your on-going help.


[main] Upgrade to latest dependencies

Cron -knative-prow-robot
/cc knative/operations-wg-leads
/assign knative/operations-wg-leads
Produced by: knative-sandbox/knobots/actions/update-deps
Details:
bumping knative.dev/eventing 5df47e1...9b9cdee:
  > 9b9cdee Adding code of conduct to reference knative/community's CoC (# 6281)
bumping knative.dev/serving dd10cc5...646d0b1:
  > 646d0b1 removing deprecated serving.knative.dev/release label (# 12754)
  > 40c3e33 upgrade to latest dependencies (# 12764)

/test pull-knative-operator-upgrade-tests

/test pull-knative-operator-eventing-upgrade-tests

/retest


[mod_openssl] ignore client verification error if not enforced

Hi,
Here is the change with every commits squashed in a single commit, as discussed here.
Regards,
Gaël

Thanks.  That diff is much more readable.

Oops, I pushed a new commit right now... have you already merged it to master?

I did push to master, though for some reason the commit hook did not push back to github.  It's on git.lighttpd.net.

I did a minor modification to the verify callback.
I reused the err_cert in the if statement.
If it is not too late, you may consider this improvement.
Sorry about that

What is the benefit of that change?  (Yes, it can be made if it makes sense.)
The patch I took (the earlier patch) followed the example in https://wiki.openssl.org/index.php/Manual:SSL_CTX_set_verify(3)#EXAMPLES and your new commit differs from the example.  Would you please explain why?

This benefits are:

it reuses the variable instead of calling again a function
it compiles with -Werror with the latest version of openssl

I tested with valgrind and there is no leaks and no errors with the latest version of openssl (1.1.0-dev).
==9622==
==9622== HEAP SUMMARY:
==9622==     in use at exit: 1,736 bytes in 7 blocks
==9622==   total heap usage: 11,208 allocs, 11,201 frees, 889,723 bytes allocated
==9622==
==9622== Searching for pointers to 7 not-freed blocks
==9622== Checked 186,016 bytes
==9622==
==9622== LEAK SUMMARY:
==9622==    definitely lost: 0 bytes in 0 blocks
==9622==    indirectly lost: 0 bytes in 0 blocks
==9622==      possibly lost: 0 bytes in 0 blocks
==9622==    still reachable: 1,736 bytes in 7 blocks
==9622==         suppressed: 0 bytes in 0 blocks
==9622== Reachable blocks (those to which a pointer was found) are not shown.
==9622== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==9622==
==9622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==9622== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

it reuses the variable instead of calling again a function

I don't see that in the patch below.
--- a/src/mod_openssl.c
+++ b/src/mod_openssl.c
@@ -237,6 +237,5 @@ verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
                           err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT)) {
-        X509_NAME_oneline(X509_get_issuer_name(ctx->current_cert), buf,
-                          sizeof(buf));
+        X509_NAME_oneline(X509_get_issuer_name(err_cert), buf, sizeof(buf));
         log_error_write(srv, __FILE__, __LINE__, "SS", "SSL: issuer=", buf);
     }

it compiles with -Werror with the latest version of openssl

I did not test with latest openssl.  What error existed prior to your latest patch?
However, I did accidentally let you insert two literal tabs ('\t') in your patch, in a .c file without literal tabs, so I plan to remove them.  Please review the diff of your latest patch with what I committed to  	04d510a in https://github.com/gstrauss/lighttpd.1.4
Are there any differences other than what I posted above?

I don't see that in the patch below.

Right, I just reuse the variable err_cert instead of ctx->current_cert which leads to a mismatch type with the newer openssl version (1.1.0).
mod_openssl.c: In function ‘verify_callback’:
mod_openssl.c:239:51: error: dereferencing pointer to incomplete type ‘X509_STORE_CTX {aka struct x509_store_ctx_st}’
         X509_NAME_oneline(X509_get_issuer_name(ctx->current_cert), buf, sizeof(buf));
                                                   ^~

I am fixing the tab and push force on this branch.

Instead of force-push, please add a new commit so that I can more easily cherry-pick it.  Thanks.

a new commit on top of 04d510af20bdff46a99ac0f800aa497636d8a19b?

Just looked at the openssl code.  Now i see that err_cert = X509_STORE_CTX_get_current_cert(ctx) is the same result as ctx->current_cert, though that structure is private in openssl 1.1.0.
Thank you for testing with openssl 1.1.0 and for the rest of your thorough testing.
Still, X509_STORE_CTX_get_current_cert() is not present in openssl 1.0.1 (different than 1.1.0), but that interface appears in 1.0.2.  Therefore, we need to use ctx->current_cert on versions earlier than openssl 1.0.2.
Also, the openssl manual page notes:

In versions of OpenSSL before 1.0 the current certificate returned by X509_STORE_CTX_get_current_cert() was never NULL. Applications should check the return value before printing out any debugging information relating to the current certificate.

i need to test this with 1.0.1 before pushing to master:
--- a/src/mod_openssl.c
+++ b/src/mod_openssl.c
@@ -219,10 +219,16 @@ verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
     }

     if (preverify_ok) {
-       return preverify_ok;
+        return preverify_ok;
     }

+  #if OPENSSL_VERSION_NUMBER >= 0x10002000L \
+   && !defined(LIBRESSL_VERSION_NUMBER)
     err_cert = X509_STORE_CTX_get_current_cert(ctx);
+  #else
+    err_cert = ctx->current_cert;
+  #endif
+    if (NULL == err_cert) return !hctx->conf.ssl_verifyclient_enforce;
     X509_NAME_oneline(X509_get_subject_name(err_cert), buf, sizeof(buf));
     log_error_write(srv, __FILE__, __LINE__, "SDSSSDSS",
                         "SSL: verify error:num=", err, ":",
@@ -235,8 +241,7 @@ verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
      */
     if (!preverify_ok && (err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY ||
                           err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT)) {
-        X509_NAME_oneline(X509_get_issuer_name(ctx->current_cert), buf,
-                          sizeof(buf));
+        X509_NAME_oneline(X509_get_issuer_name(err_cert), buf, sizeof(buf));
         log_error_write(srv, __FILE__, __LINE__, "SS", "SSL: issuer=", buf);
     }

@@ -574,7 +579,7 @@ network_init_ssl (server *srv, void *p_d)
         }

         if (NULL == s->ssl_ca_file_cert_names
-           && !buffer_string_is_empty(s->ssl_ca_file)) {
+            && !buffer_string_is_empty(s->ssl_ca_file)) {
             s->ssl_ca_file_cert_names =
               SSL_load_client_CA_file(s->ssl_ca_file->ptr);
             if (NULL == s->ssl_ca_file_cert_names) {

A modification needs to be made to what was committed.  X509_NAME_oneline() is deprecated.
The manpage for X509_NAME_oneline() says:

The functions X509_NAME_oneline() and X509_NAME_print() are legacy functions which produce a non standard output form, they don't handle multi character fields and have various quirks and inconsistencies. Their use is strongly discouraged in new applications.

Besides X509_NAME_oneline() function being deprecated, until fairly recently, there was a security issue with the function, too.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-2176

The X509_NAME_oneline function in crypto/x509/x509_obj.c in OpenSSL before 1.0.1t and 1.0.2 before 1.0.2h allows remote attackers to obtain sensitive information from process stack memory or cause a denial of service (buffer over-read) via crafted EBCDIC ASN.1 data.

Please consider using X509_NAME_print_ex() and propose a reasonable set of flags for a consistent and still-useful result.
The above was also noted in https://redmine.lighttpd.net/issues/2693 and related pull request https://github.com/lighttpd/lighttpd1.4/pull/63

Okay.
I will check for that.
I am currently on holliday, but I will find a moment to fix and test this issue.

Enjoy your holiday.  I have what I think is a solution, and I plan to test and push tomorrow.


Future of torchdata and dataloading

As of July 2023, we have paused active development on TorchData and have paused new releases. We have learnt a lot from building it and hearing from users, but also believe we need to re-evaluate the technical design and approach given how much the industry has changed since we began the project. During the rest of 2023 we will be re-evaluating our plans in this space.
We want to hear from users on their use-cases and the pain-points they have (with data loading in general or torchdata specifically). Please reply on this issue to help inform our future roadmap.

Thanks for the update, @laurencer. Does this mean torchdata as a domain library or the entire concept of datapipes/DataLoader2 is under review?

The short answer is we need to look at both. More holistically there's lots of benefits with datapipes & DataloaderV2, however we've seen some limitations in a few use-cases which indicate we may need to tweak them a bit (or they're not the one-stop solution that we were hoping for). Overall the data loading space is really important and we hear about a lot of pain-points so we want to make sure we get the core abstractions right.

I too would like to hear what limitations you are referencing.
If it is performance oriented, I believe there's an argument. You could make something compatible with a compiled framework, especially something like torch.compile. Someone recently talked about speedups loading tar files with rust as an example.
My question is what you recommend as an alternative platform to torchdata for flexible and fast dataloading in pytorch?

Does the design re-evaluation apply only to TorchData, or also to the portion of the datapipes API that was upstreamed to PyTorch core?
https://github.com/pytorch/pytorch/tree/main/torch/utils/data/datapipes

IMO the first thing that comes to mind is TensorFlow's tf.data.Dataset API, which is super cool to use. It is deeply integrated in the framework and with Keras, and the different operations you "pipe"/chain together can be fused at runtime so that the input pipeline is more optimized.
However there are still some solutions such as NVIDIA DALI that is waaay faster if it applies to your use case.
I don't know TorchData in much details but I'd say building a nice-looking pipeline is easy, but making a pipeline optimized for high performance while still making it look cool is the real challenge.
By cool-looking I mean nice/easy to read as code, but also easily extensible, like users could share "databricks" together or something.
I hope this helps!

Also, do you still recommend to use torchData in the meantime or will the compatibility with torch break at some point so we should avoid using it?

Our ML team have been avid users of torchdata. We have used it to build datapipes that fetch large raster datasets from cloud providers to support training and inference. Currently we have a couple projects using torchdata, the most robust is
https://zen3geo.readthedocs.io/en/latest/walkthrough.html managed a library by my colleague @weiji14 for fetching and batching large satellite images, with steps organized as functional datapipe ops.
While the API makes it easier to reuse custom data operations, we've been running into some consistent pain points when integrating datapipes with Dataloader V1 or Dataloader V2. We've had to switch back to Dataloader V1 and Datasets.
It doesn't seem like there is a clear  set of documented rules for prefetching, shuffling, buffer sizes, memory pinning that results in good performance, or even any performance gains that beats single process dataloading when using torchdata with either Dataloader. All the configurations we have tried result in hangups, out of memory errors, or slower performance than a single process. It's also unclear how these parameters interact with different reading services.
I would love to see better docs and functionality for setting prefetching, shuffling, etc. with different kinds of reading services. Being able to profile datapipes and inspect RAM and cpu consumption of each operation would also be invaluable.

I started to prototype a new version of my event data library on top of torchdata. The API is very clean and easy to understand, which is a strong plus point, even if it has a minor performance impact (I didn't verify that). I remember struggling with DataLoader2 and making multithreading work.

Hey, thanks for the update. Does that mean that torchdata will become obsolete in the future?
As I already indicated in older issues, what I see as the biggest weakness right now, is the lack of control and flexibility with regards to:

Shuffling
Sharding
Multiprocessing (dispatching to other processes etc.)

These things are right now tightly integrated into the torchdata core, and not easily accessible from user code. Giving User code the same Power and flexibility is paramount in my opinion to facilitate more complex pipelines than the vanilla cv pipeline. Or contrary, these functionalities should be implemented in user land without privileged handling from torchdata.
You can have a look at my repository (sehoffmann/atmodata) for some monkey patches that were necessary to facilitate my pipeline.

As for me, the ideal data pipeline should be ergonomic, flexible and efficient.
Chainable iterator already shows its power in iterator algorithm libraries like itertools and more-itertools. torchdata choose to enhance that with a functional programming API, that's good.
But flexibilty still has room for improvement. At least, torchdata should be comparible to pypeln which has good flexibility to switch or mix thread/process/coroutine based tasks.

For performance, can torchdata be comparible with huggingface dataset? Can we easily leverage arrow or something else to build a high-performance data pipeline? It's better to show more benchmarks on production data.

DGL team is currently studying what should the UX be for scaling deep learning on graphs, namely the sampling strategies.  More specifically, we want to support customization on

Different graph storages (in-memory, disk, graph databases, etc.)
Different node/edge feature storages (in-memory, disk, etc.)
Sampling algorithms (online neighbor/subgraph sampling, offline sampling, etc.)
Downstream tasks (node classification, link prediction with negative sampling, graph classification, etc.)
Orchestration (whether to put sampling and feature fetching in multiprocessing/multithreading, how to schedule different stages, etc.).

So our current design depends on the composability of torchdata's DataPipes to allow for maximum extensibility, expressing the graph storages/feature storages/sampling algorithms/etc. as a composition of iterables and their transforms.
That being said, we are currently not pursuing active usages on DataLoader2 due to concerns on compatibility to existing packages depending on PyTorch DataLoader (e.g. PyTorch Lightning).  That being said, we borrowed some ideas from ReadingService (namely the in-place editing of DataPipes).
We already have some demo in https://github.com/dmlc/dgl/tree/master/tests/python/pytorch/graphbolt.
Happy to discuss further.

@npuichigo pypeln looks interesting. Based on there being multiple single-threaded queues between stages I assume this is designed for a single-node setup? PyTorch users would need multi-node support. To insert a queue between stages of a pipeline with multi-node stages, presumably we'd want to use some kind of purpose-built stand alone message queue. I'm not sure if that kind of setup is desirable -- once the training data reaches GPU hosts, I'd think we usually don't want to send it back elsewhere, so that architecture might make more sense for pre-processing.
@BarclayII

our current design depends on the composability of torchdata's DataPipes
not pursuing active usages on DataLoader2

I'd like to use DataPipes for some NLP problems for similar reasons, and have some prototypes. I'd like to get confirmation, but from what I can tell it seems like it may only be TorchData that has paused development, whereas the DataPipes API is already part of PyTorch core.
For my use-cases I don't need DataLoader2 or readingservice/adapter. I think there are other ways to solve the problems addressed by those additional APIs -- I think there are ways to do it with just DataPipes that would address the concerns @sehoffmann raises around shuffling/sharding being "tightly integrated into the torchdata core, and not easily accessible from user code."
I also wonder whether we actually need a separate API focused on composability of datasets. The original dataset API could be used with composition too, and I'm not sure exactly what challenges we'd face in doing so. I know we wouldn't have the functional helper functions but that seems minor, and not sure what else we'd be missing.

however we've seen some limitations in a few use-cases which indicate we may need to tweak them a bit

@laurencer can you elaborate on which are those use-cases?

@npuichigo I checked pypeln as well.  It seems that the user needs to specify how to organize the queues in low-level (e.g. multiprocessing, multithreading, asyncio, etc.).  Normally our UX shouldn't involve such a low-level specification unless the developers want to implement their own pipeline scheduling.

The original dataset API could be used with composition too, and I'm not sure exactly what challenges we'd face in doing so. I know we wouldn't have the functional helper functions but that seems minor, and not sure what else we'd be missing.

@nairbv Other than the functional helper functions, I find the in-place editing of DataPipe useful.  For instance, with a single-processing DataPipe. I can have a DataLoader that changes the DataPipe for multi-processing and the process will be transparent to users.  We also intend to apply the same idea for coordinate graph sampling, feature prefetching, and CPU-GPU transfer.  https://github.com/dmlc/dgl/blob/master/python/dgl/graphbolt/dataloader.py#L58 shows an example.
Happy to discuss further.

As for data pipelining solution, it would be nice if this could be developed without a dependency on a deep learning framework (torch, tensorflow etc),

Thanks for the update @laurencer Laurence.
This is unfortunate news to hear as we over here at Microsoft's Developer Experience team have seen a lot of interest in cloud computing and seamless integration between Azuree Storage and PyTorch from our customers. Our summer intern's project was building out a custom FileLoader and FileLister DataPipe that allowed easily interacting with datasets that are stored on Azure Storage, and so the news of a halt in development and an uncertain future of the torchdata repo makes for a difficult situation in regard to planning our future in terms of continuing development of integration with PyTorch.
With that being said, I am hopeful that this is a necessary step back in order to re-strategize and refine the future roadmap to ultimately end up with a better user experience for all. As the field of AI/ML continues to develop in the near future, it is really a matter of when (and not if) we will revisit building direct support for Azure Storage with PyTorch workflows, and so we will likely reach out sometime when the future of torchdata and dataloading is clearer. However, there are still some questions (and many were raised by previous posters above) that I would like to echo which would greatly help us developers in the interim of no new releases:

What are the expectations of compatibility / should we even consider continuing to build out torchdata support, or is that something that is becoming obsolete in the near future?
Any ETAs for when we will be updated on the status of the future of torchdata? While I understand this may be difficult, it would be great if any information could be shared in regard to timing to keep any developers from continuing to build out their infrastructure with torchdata, or if they should begin the migration process etc.

Thanks in advance, and please feel free to reach out if necessary!

good material https://arxiv.org/pdf/2211.04908.pdf

What are the expectations of compatibility / should we even consider continuing to build out torchdata support, or is that something that is becoming obsolete in the near future?
Any ETAs for when we will be updated on the status of the future of torchdata? While I understand this may be difficult, it would be great if any information could be shared in regard to timing to keep any developers from continuing to build out their infrastructure with torchdata, or if they should begin the migration process etc.

+1, any updates on the roadmap since September?

I found an alternative to use Ray Data for data loading. It's framework-agnostic and performant, also providing a chainable API but with more fine-grained parallism control. I also write a tutorial to use together with HuggingFace Dataset here https://github.com/npuichigo/blazing-fast-io-tutorial.

I'd like to share our work https://github.com/google/space that supports materialized views in ML data pipelines. It ingests more metadata (versioning, column stats, logical plan) to ML datasets and pipelines, to provide a database/lakehouse like experience. Materialized views have the benefits of incrementally processing (and go back to old versions) and data lineage. Hope it will be useful.

I am also interested about the torchdata and datloading vision/roadmap related to audio/video api fragmentation https://github.com/pytorch/pytorch/issues/81102

Any update here?  My team is interested in creating an Azure Storage extension for PyTorch, similar to S3, but we cannot proceed with planning and implementation until we know what the future of PyTorch extensions will look like.

I don't know if @laurencer is still active on this or he is working for  pytorch/Meta more in general. /cc  @ejguan as he is the codeowner of pytorch dataloader.

@bhack I know @ejguan isn't still on torchdata (I think he's in ads now)

@nairbv I think that the codeowners need to be updated https://github.com/pytorch/pytorch/blob/main/CODEOWNERS#L113-L114

Agreed. I don't believe dataloader currently has a specific owner but I'm not still at Meta.

I hope it will be one of the topics at https://events.linuxfoundation.org/pytorch-conference/

Hi everyone, we’d like to share an update about how we plan to use the pytorch/data repo going forward. We will be focusing our efforts on a more iterative approach to Dataloader v1 (torch.utils.data), and we plan to use torchdata as a place to iterate on new features, such as the StatefulDataLoader. To avoid confusion in our support and offering, we will be deleting DataLoader2 and DataPipes from this repo. We plan to restart the torchdata release cycle, beginning with 0.8.0 in July 2024 (following PyTorch 2.4.0). DataLoader2 and DataPipes will be marked deprecated. In the October 0.9.0 release, they will no longer be part of the torchdata package, and existing users will need to either migrate away or pin to an older release such as 0.8.0. We welcome feedback to these plans, and will be responding in this Issue to any questions and concerns.

Is it possible to publish some whitepaper / vision showcasing where the current steering committee's thought is, and consider an RFC process?

Thanks for the update @andrewkho!

Are there any places we learn more about the roadmap? A design doc for StatefulDataLoader or anything we can use to help inform our decisions on how to engineer future-oriented performant data pipelines? We've been in a bit of a holding pattern waiting for clarity before needing to sink tons of eng hours on reimplementing complex data pipelines...
What does the removal of DataPipes from this repo mean for the corresponding implementation in core torch (torch.util.data.datapipes)?

@andrewkho Glad to hear work will restart on this project.  The plan fails to explain what they are being replaced with.
- StatefulDataLoader looks like a very small part of a much bigger picture.
- It would be helpful give a plan / vision of that is going to proceed.
- It would be helpful to know what was the original problem with torchdata/datapipes so other's can learn/avoid whatever antipatterns/scalability issues/API issues caused this project to get paused and completely redone.

It would be nice in this roadmap also to consider how to coordinate consolidate with some other projects in the ecosystem for their specific data domain. See https://github.com/pytorch/pytorch/issues/81102

Thanks everyone for the questions and comments. I'd like to acknowledge @josiahls 's comment that we are removing stuff without saying what we're replacing it with, which is completely fair criticism. We want to get feedback and see if there is significant pushback on something that we are sure about (deleting datapipes/dataloader2) as soon as possible.
Regarding roadmaps and plans, we're currently iterating on some ideas on how best to serve the community, but it's not quite in a state that is ready to discuss yet. I'll be able to share more in the next couple of months, please bear with us as we get our act together :)
Re: datapipes in torch.utils.data, we're going to be taking a hard look at that as well.

Thanks @andrewkho! We'll be looking forward to hearing more.
Just to +1 what has been pointed out before in this thread: the tf.data.Dataset is a majorly performant system to consider, allowing you to build IterDataPipe-style chains of transformations, each with autographed functions (tf.function) and auto-tunable level of parallelism. It's awesome to be able to control concurrency at each part of the data pipeline, and it makes for super composable data pipeline blocks.

There are a couple of different approaches to stateful / savable datasets in IBM repos that may be of interest. These have been used to stop and restart long-running training jobs when training LLMs.
SavableDataset: https://github.com/foundation-model-stack/foundation-model-stack/blob/main/fms/datasets/util.py#L65
StatefulDataset: https://github.com/foundation-model-stack/fms-fsdp/blob/main/fms_fsdp/utils/dataset_utils.py#L68

It's understandable to remove datapipes from this repo, as it is just a pass through to the torch version. I do like the functional datapipes approach, allowing for dp.someiterable nomenclature. I hope it can be ported.
For data loader 2, is there a concrete idea for how to do a data sampler approach in dl1? I typically don't have a length in iterable datapipes, or they are infinite cycles. I'm unaware if a standard was created. The analogous reading services for distributed and multiprocessing are very nice here.
Glad to hear it is coming back in some form. To me, it is a useful repo for datapipes and accessing cloud providers (e.g. S3). I wonder what the big vision is after removing dp and dl2 🤣

We have been using dbt: @dbt-labs helps us transform our pre-train data and upload it automatically to S3 for streaming - I think a similar pipeline should be feasible for DataLoader2-type tasks? Not sure...
Here's an example for health care data: https://github.com/onefact/healthcare-data/

StatefulDataLoader may be a prerequisite for stateful data pipes.

Hi everyone, our roadmap for the second half of the 2024 is available publicly alongside many other PyTorch efforts and I wanted to share this here as well in case you haven't seen it yet. The KRs are subject to change as we learn more, and there is still more work to do to get clarity on the overall effort. If you have feedback or thoughts, we're very interested to hear from you folks :)
Also another update, we'll be at PyTorch Conference in San Francisco this September, if anyone is attending we'd be happy to meet up and talk about anything!

@andrewkho Thanks for the update! So if I am understanding correctly:
The replacement for torchdata datapipes themselves might look something like:

huggingface et al. datasets (which actually are built on apache arrow what it looks like?)
torchdata will attempt using learnings from what huggingface et al have implemented and move the basic functionality into a core API.
Problem with original datapipe API was it did not get buy in from the larger OSS space.

Using huggingface API as an example:
I have only worked a little bit with huggingface datasets. From going through their documentation and my memory working with them. Maybe someone more knowledgeable can enlighten me:
Pros:

Huggingface dataset API is intended as a scaffold for custom pipelines.
Pretty good docs.
Core methods / functions / processes being just methods of a common object are easier to step through when debugging compared to datapipes.
Custom functions such as transforms are just input / output callable.

Cons (could be just ignorance of what hg datasets can do):

Custom behaviors for shuffling / sharding / dataloading / mapping can only be done via inheritance. Personal taste inheritance is hard to reason with when objects have many abstractions.

Was a fan of the datapipe API being very horizontal  The question of "what does this pipe do?" could be answered (generally) on a single page and a single object, as opposed to jump through several hierarchies or scrolling for 1000s of lines of code.

Transforms with internal state / params would need to be separate callable objects that are instantiated separately and then passed to the .map. This is as opposed to just using a custom datapipe
In general "scaffolds" can be limiting, and trying to customize them isn't fun. Datapipes in the end were just iterators, so doing "weird" stuff was more straightforward.
Played around with making an RL framework in torchdata which was doable since the API itself was pretty horizontal.

This the cons aren't a condemnation of the decision to look to HG et al though since they are popular frameworks for a reason. I'm looking forward to see how torchdata leverages their work and make it more flexible. As for me personally, I'll also look at HG and try out their API (for transforms / multimodel / RL training) see if the cons above aren't really cons at all.

@josiahls we definitely do not want to force users to take dependencies on HuggingFace which is more application focused. The focus is more that we want to ensure we're compatible and working together with them, since they depend on PyTorch and LLM fine-tuning is such an important use case today, but does not encompass everything that the OSS community is working on. Hope this helps

@andrewkho Please also take litdata into consideration

@andrewkho Are the datapipes in torch.utils.data going to be removed as well?

Are you going to remove the datapipe related code from torch.util.data? If so, can you announce the plan in advance and give us several months to take action?

I'd like to chime in and say that it is very unfortunate that the official online documentation make almost no mention of this planned deprecation. In fact, I only was made aware of this planned of this issue this morning when I updated my version of torchdata.
Adding a banner warning on of deprecation and linking to this github issue would better communicate to ML developers the current status of this project. Keeping the official documents up to date with the current project status is an important issue as otherwise developers may be unaware and waste crucial time and effort into projects that will not be supported in the future.

as a user of torchdata, i was very happy to see the resurrection of the project.
i have a question about the development plan. from the README, i see:
torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader
this is somewhat surprising. although the current Datapipes seem to have various issues underneath the shell, so far, Datapipes ARE torchdata. the current API reference:
API Reference:
Stateful DataLoader
Iterable-style DataPipes
Map-style DataPipes
Utility Functions
DataLoader2
ReadingService
and this is it; i.e., until ver 0.7, torchdata == the datapipes and other necessary utilities (dataloader2 and reading service).
and that's why it is surprising for me, that while the development of torchdata has re-started, it is being done in a way it discards everything it had.
so, can i ask for a bit more details about what the new direction (enhancement of torch.utils.data.DataLoader)? or am i missing something here?
thanks.

to add more context: i love datapipes and want them improved, not deleted.

although alternatives seem to exist, none of these (HuggingFace Datasets, WebDatasets, Mosaic, Ray, Nvidia DALI) are a general-purpose, scalable data loading solution that is native to torch i.e., expected to be compatible with other libraries such as Accelerator, Lightning, etc. E.g., MosaicML dataset is not compatible with Lightning's multi-node training. in short, none of them enabled us to load multimodal data from remote (s3 buckets) with multi-node training. datapipes are the only working solution. even if they are somehow compatible in some cases, they are not expected so. they can always become incompatible with something they're not responsible for.
why then the low adoption of datapipes? i believe, perhaps for these reasons, based on my experience.

lack of documentation and examples.
difficulty in debugging and profiling.

when it's slow, it's really hard to figure out why
with some unexpected timeout, i had to monitor & re-start the training quite often.

lack of automatic optimization such as tf.data.AUTOTUNE (yes.. tensorflow..)

even so, i still believe

the API is great! functional datapipes for loading and .map() with a distributed setup.. what else?
100% that this should be done on torch, natively
improving it would lead to much happier and wider adoption.

I'd like to chime in and say that it is very unfortunate that the official online documentation make almost no mention of this planned deprecation. In fact, I only was made aware of this planned of this issue this morning when I updated my version of torchdata.
Adding a banner warning of deprecation and linking to this github issue would better communicate to ML developers the current status of this project. Keeping the official documents up to date with the current project status is an important issue as otherwise developers may be unaware and waste crucial time and effort into projects that will not be supported in the future.

Thanks @KBlansit for calling out the gap in notice in that document, we'll be updating that this week. cc @gokulavasan

as a user of torchdata, i was very happy to see the resurrection of the project.
i have a question about the development plan. from the README, i see:

torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader

this is somewhat surprising. although the current Datapipes seem to have various issues underneath the shell, so far, Datapipes ARE torchdata. the current API reference:

API Reference:
Stateful DataLoader
Iterable-style DataPipes
Map-style DataPipes
Utility Functions
DataLoader2
ReadingService

and this is it; i.e., until ver 0.7, torchdata == the datapipes and other necessary utilities (dataloader2 and reading service).
that's why it is surprising for me, that while the development of torchdata has re-started, it is being done in a way it discards everything it had.
so, can i ask for a bit more details about what the new direction (enhancement of torch.utils.data.DataLoader)? or am i missing something here?
thanks.

Hi @keunwoochoi , thanks for the feedback! Unfortunately we aren't quite ready to share any more details about the new direction quite yet (we are actively iterating on this), but we do really appreciate the feedback and will update this thread as soon as we can.

I side with @keunwoochoi in feeling quite enthusiast when I discovered that torchdata was finally getting some updates only to discover the planned disappearances of datapipes.
While not perfect (and I share the grievances around how difficult debugging and profiling can get with datapipes), I did not found an equivalent composable solution with near unlimited flexibility (notably through the zip/unzip and map operators). This allows to prototype and tests new idea whilst virtually ignoring the implementation side and instead focus on actual data science work which is why I am getting payed in the first place.
For me, DataPipes are an asset of the PyTorch ecosystem, not a historical glitch that should be discretely covered.
For what it's worth I do believe that a significant reason for the lack of adoption was the relative confidential nature of the API. Although it enjoyed some publicity in release messages, it kept getting advertised as "beta" thus discouraging users to invest in learning the solution. Paired with the scarce documentation and examples mentioned before, this made moving an existing code base from the monolithic Dataset API to DataPipe was quite the endeavor, but a rewarding if you managed to finally get there.
With that in mind I can only press you to improve and double down on DataPipes rather that removing them entirely.

I share the same feeling as @keunwoochoi and @ClementMaliet .
DataPipe's composability is why the GraphBolt subpackage from DGL (https://github.com/dmlc/dgl) chose it in the first place.  Scalable training with graph neural networks can have very diverse graph storage, feature storage, algorithm, and computing device.  And DataPipe's composability allows users and developers to easily adapt a graph sampling algorithm or deploy optimizations by focusing on the most important part only, whether it being one particular device, algorithm, or optimizing the entire DataPipe graph execution with different scheduling and parallelism strategies.
cc @frozenbugs .

Same here as @keunwoochoi, @ClementMaliet and @BarclayII.
One more thing. DataPipe as DAG maps well to a structured config, and gives user full flexibility to define data loading logics without having to write boilerplate codes.
We heavily depends on DataPipe and hydra to dynamically construct data pipeline for end users. For example, users may want to construct a balanced datapipe for training, it's simple to use a hydra yaml config like:
dataset:
  _target_: torchdata.datapipes.iter.Batcher
  batch_size: 32
  datapipe:
    _target_: torchdata.datapipes.iter.Multiplexer
    _args_:
    - name: animal
      _target_: CustomSampleMultiplexer
      _args_:
      - weight: 0.7
        pipe:
          _target_: CustomImageFolder
          path: /path/to/dog
      - weight: 0.3
        pipe:
          _target_: CustomImageFolder
          path: /path/to/cat
    - name: plant
      _target_: CustomSampleMultiplexer
      _args_:
      - weight: 0.5
        pipe:
          _target_: CustomImageFolder
          path: /path/to/fern
      - weight: 0.5
        pipe:
          _target_: CustomImageFolder
          path: /path/to/moss

Thanks everyone for the comments and feedback on datapipes, we're planning to have something that will be composable (eg with Hydra configs) and will provide migration guides for those coming from datapipes. I'll share more details for feedback as soon as I am able.

Hi everyone, I've just posted this issue with details on what we've been thinking about as the successor to DataPipes. Please leave feedback specific to the RFC on that issue's comments: https://github.com/pytorch/data/issues/1318. Also a few of us will be at PyTorch Conference this week, hoping to connect with as many of you as we can!

@andrew-bydlon Thanks for the RFC link! Is there a timeline to release a stable successor to DataPipes? I have a project which needs a streaming dataloader right now and am trying to understand the recommended short-term approach. Do I just pin old versions of torch / torchdata, use the deprecated features for now, and migrate later?

Our current plan / timeline as follows:

0.9.0 stable release WITH datapipes/dlv2 to go out later this month following PyTorch 2.5.0
After 0.9.0 (on main): delete datapipes/dlv2 and start landing code related to RFC, also in October
Subsequent releases (0.10.0+) will have successor code but no datapipes/dlv2

@addisonklinke Are you interested in testing out the datapipes successor?

I found the archive datapipes (tar, tfrecord) very handy as it is often convenient to move archived data from one server to another. Will these datapipes logic be ported onto the datapipes successor ?

@gdoras yes we'd like to port over those functions as well. If you're interested in contributing, let's discuss what would be most important for you to knock out first

What is the suggested alternative now to strram data efficiently like from tfrecords for pytorch training?

@merryHunter For now datapipes are still available in the latest stable release, but we'll be adding a tfrecord reader at some point soon to the new torchdata.nodes API

Update: 0.10.1 was just released which completes the deletion of datapipes and dataloader2. torchdata.nodes replaces them. We do not yet have 1-1 feature parity with datapipes yet, so until then (some time in 2025), please pin to torchdata=0.9.0 and let us know about the missing functionality. Or even better, consider contributing!
https://github.com/pytorch/data/releases/tag/v0.10.1


problem: force to use USB audio card won't work.

Hi. I have Raspberry Pi OS installed with a docker container containing Home Assistant.
The problem is that the home assistant container overwrites Raspotify audio output and I don't hear nothing when I connect my USB audio card.
Is there a way to disable docker audio and force original source audio output from Raspotify?
Thanks

You need to file an issue with Home Assistant or where ever you got the docker container from and ask them how to prevent it from breaking audio for Raspotify.


Why correctness value is low even if model responded correctly?

Ground truth- Tag 82 represents the Application Interchange Profile (AIP).
LLM Response- In EMV contactless transactions, tag 82 represents the Application Interchange Profile (AIP). The AIP is a set of persistent data elements representing the context of an ongoing online transaction. The Online Transaction Context is saved by the Kernel before returning the Online Request. The AIP is identified by the tag '82' in the Static Data to be Authenticated (item 9d of Table 6.5). If the SDA Tag List is present
Answer Correctness- 0.437377148125797
Answer Similarity- 0.892365735360333

What is the acceptable range of answer correctness score?

No, I mean what correctness value is accepted for a QA accuracy on a context based datase?


hello there is a minus problem when running the code, could you please help? thanks

the first file, called "crawler_reuters.py", it shows up:
**Traceback (most recent call last):
File "", line 8, in
main()
File "", line 2, in main
arg = sys.argv[2]
IndexError: list index out of range**
how can I fix it? Thanks

Are you sure you are using the most-updated version of crawler_reuters.py? Because in that version, I can't see any line with sys.argv[2]

Hi, @Yuminzhou , I changed the framework to a new version, it is much more powerful and easier to understand, you are welcome to take a look at it.


BoltGraphClient  transaction does not work  when execute query that invoke method ExecuteWithoutResultsAsync() in transaction scope.

Hi,
I have ran into the following issue: when I execute query in BoltGraphClient transaction scope :
var graphClient= new BoltGraphClient(new Uri("bolt://<IP_ADDRESS>:7687"),
                "neo4j",
                "neo4j");
client.Connect();

ITransactionalGraphClient txGc = (ITransactionalGraphClient) graphClient;
 using (var tx = txGc.BeginTransaction())
     {
           txGc.Cypher.Match("(n)").Set("n.Value = 'test'").ExecuteWithoutResultsAsync();
           tx.Commit();
     }

the transaction hangs and query never executed.
I have looked into IRawGraphClient.ExecuteCypherAsync method implementation  and found that
problem could easily by solved by adding else statement after check about "InTransaction" invocation. In the IRawGraphClient.ExecuteGetCypherResultsAsync method there is the same check.
This is my sovle for the problem:
async Task IRawGraphClient.ExecuteCypherAsync(CypherQuery query)
        {
            var tx = ExecutionContext.Begin(this);
            string lastBookmark = null;

            if (Driver == null)
                throw new InvalidOperationException("Can't execute cypher unless you have connected to the server.");

            if (InTransaction)
            {
                await transactionManager.EnqueueCypherRequest($"The query was: {query.QueryText}", this, query).ConfigureAwait(false);
                OperationCompleted?.Invoke(this, new OperationCompletedEventArgs
                {
                    QueryText = $"BOLT:{query.QueryText}"
                });
            }
            else
            {

                using (var session = Driver.Session(query.IsWrite ? AccessMode.Write : AccessMode.Read, query.Bookmarks))
                {
                    if (query.IsWrite)
                        await session.WriteTransactionAsync(async s => await s.RunAsync(query, this)).ConfigureAwait(false);
                    else
                        await session.ReadTransactionAsync(async s => await s.RunAsync(query, this)).ConfigureAwait(false);
                    lastBookmark = session.LastBookmark;
                }
            }

            tx.Complete(query, lastBookmark);
        }

Could you please advise?

Hi,
Feel free to submit a PR with this in, and a suitable test if you can - and we'll take it from there

As TransactionScope is no longer in use in the 4.x version of the codebase, I am going to close this - if it's still a problem feel free to reopen.

As TransactionScope is no longer in use in the 4.x version of the codebase, I am going to close this - if it's still a problem feel free to reopen.


Any way to find out if MediaPipe is currently detecting a Hand Pose?

yes https://github.com/keijiro/BlazePalmBarracuda/blob/main/Packages/jp.keijiro.mediapipe.blazepalm/Runtime/PalmDetector.cs#L33

Hey that was lighting fast. Thanks, Keijiro. There is a nuance though. detections.Length returns as 0 if a part of the hand is obstructed (for example if my palm was outside the camera frustum). A confidence score would be nice just to see if I'm catching anything, but thanks for your pointer to begin with.
Here's my logic:
    void Start()
    {
        _pipeline = new HandPipeline(_resources);
        _palmDetector = new PalmDetector(_resources.blazePalm);
    }
    void LateUpdate()
    {
        var detections = _palmDetector.Detections;
        bool isHandDetected = detections.Length > 0;

        if (isHandDetected)
        {
            Debug.Log(1);
        }
        else
        {
            Debug.Log(0);
        }
   }


PixelShader debug: Remove line numbers

Description
Add option to remove/hide line numers in code, or at least, when selecting text ignore them?
Environment

RenderDoc version: 1.10
Operating System: W10 x64
Graphics API: D3D11

As outlined in the documentation can you please explain what your ultimate end goal is and why you need this?

not necessary remove but at least making them dummy like the ones on the left, to be more friendly to work with the code, because is too annoying removing them manually or making script to do it after you copy-paste the code.

I don't consider this change worthwhile. Copying the assembly text I don't think is a particularly common workflow as I'm not sure what you would do with it from there, and the instruction numbers are easy to remove with regexp replacement or column select in any text editor. You could also disassemble the bytecode yourself without them using fxc.


docs: deprecated Astro.props.slug → Astro.props.entry.slug

Description
While migrating some projects to Astro v5 (+ Starlight), building the website showed warning messages that Astro.props.slug is deprecated.
If I'm not wrong, Astro.props.slug can be safely replaced by Astro.props.entry.slug.
This PR applies this change in Starlight documentation in the override example.
Note: I wasn't sure whether or not I needed to modify also the translated files. Please tell me if I need also to modify the other files.
This PR doesn't contain any changesets as it's a documentation modification only.

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

en
guides/overriding-components.mdx
Source changed, localizations will be marked as outdated.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


Show the original dependency in case of a transient dependency for npm packages

Could you include the original dependency in the description of a PR in case it's a transient dependency?
More specifically for the npm ecosystem (yarn why json5). Including this information would make it faster to triage the updates based on the description alone. This is especially interesting for automatic security updates and triaging their impact on the project.

I think it's a nice idea, maybe something like

Closing as a duplicate of:

https://github.com/dependabot/dependabot-core/issues/2125


Update .htaccess

This can be a solution for resolve a cache issue. Works for me!
https://discuss.flarum.org/d/17793-a-force-refresh-fixes-everything-temporarily/49
https://github.com/flarum/core/issues/1673

Fixed in https://github.com/flarum/flarum/commit/e2544a2a223b8ab2fb9efe00036b755b6e2cd7e7


Nugetizer 300 no longer generates multi-platform projects

I downloaded the vsix, I installed it to 2017, I clicked the template to generate a multi-platform library, I get the following errors:
Blah, blah, blah, failed to install packages: 15.0_6E3324E5\EXTENSIONS\YARD5RV4.JRX\Packages:
Nuget.Build.Packaging.0.2.5.-pr172.11 : Package restore failed. Rolling back package changes for 'Library2.Nuget'

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 44cfe14b-198c-56d5-0b63-de00dd08bafe
Version Independent ID: 39eea43d-d7b5-1659-1583-dc23d099c9f8
Content: NuGet Multiplatform Library Projects (aka Nugetizer 3000) - Xamarin
Content Source: docs/cross-platform/app-fundamentals/nuget-multiplatform-libraries/index.md
Product: xamarin
Technology: xamarin-crossplatform
GitHub Login: @conceptdev
Microsoft Alias: crdun

Hi @MouthOfMadness for product bugs, please report from within Visual Studio so that we get useful product and environment information that can help diagnose the issue. Thanks.