OutputPath issue when migrate BTDF project from 5.0 to 5.5

I recently migrated a BTDF 5.0 project (BizTalk 2010) to BTDF 5.5 (BizTalk 2013) and the new project compiled without error, but it doesn't work when I deploy it. The following error occurred immediately after finishing msi installation and begin deployment. It can't event get into InstallWizard screen.
Target Clean:
C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\Framework\BizTalkDeploymentFramework.WiXSetup.targets(187,5): error : Cannot continue. The OutputPath property is undefined. By default, this property is defined in the .btdfproj in PropertyGroups conditional to Debug and Release configurations.
Done building target "Clean" in project "Deployment.btdfproj" -- FAILED.
Didn't find resolution so far, could somebody help?
Attachments
BTDF5 Issue Report.doc
This work item was migrated from CodePlex
CodePlex work item ID: '10579'
Vote count: '1'

[tfabraham@4/1/2014]
OutputPath is only used during MSI builds (the "Installer" MSBuild target), which is the default target that executes when no other target is named on the MSBuild command line.  Clearly that's not what should be running on a server.
Did you build the MSI on a build server?  If so, did you upgrade the Deployment Framework on the build server?  Do your deployments work in Visual Studio, including running an MSI build?

[xyunfan@4/2/2014]
Thanks for your quick response. I built the MSI in VS2012. This morning, I compared the MSI built in VS2010+BTDF5.0 and MSI built in VS2012+BTDF5.5 and found the batch files are missing in new MSI. That definitely explained why my build doesn't work. But, I still don't know why the batch files are missing.
Please advise how can I solve this issue.
Thanks!

[xyunfan@4/2/2014]
More Info:
I found the batch files in BTDF 5.0 are copied from:
C:\Program Files (x86)\Deployment Framework for BizTalk\5.0\Framework
However, in BTDF 5.5, there is not batch file under the folder:
C:\Program Files (x86)\Deployment Framework for BizTalk 5.5\Framework
Not quite sure if there is anything wrong there.
Please advise!

[tfabraham@4/2/2014]
If you're referring to DeployServerWizard.bat, etc. they are no longer present in v5.5.  They were called by the Start menu shortcuts.  Instead, in v5.5 the shortcuts call MSBuild.exe.
If you are scripting a deployment then I recommend that you run MSBuild.exe against BizTalkDeploymentFramework.ServerExecute.targets, which is in the Framework folder where the .bat files used to be located.
Example from the comments in BizTalkDeploymentFramework.ServerExecute.targets:
Assuming 1) BizTalk 2010/13 (/tv:4.0) and 2) default MSBuild.exe is MSBuild 4.x
Current directory is C:\Program Files (x86)\Deployment Framework for BizTalk - Advanced\1.0>
MSBuild.exe Deployment\Framework\BizTalkDeploymentFramework.ServerExecute.targets /t:Deploy /p:BT_DEPLOY_MGMT_DB=True /p:ENV_SETTINGS="C:\Program Files (x86)\Deployment Framework for BizTalk - Advanced\1.0\Deployment\EnvironmentSettings\local_settings.xml" /tv:4.0 /p:ProjectFile=Deployment.btdfproj

[xyunfan@4/2/2014]
Thanks for your instruction. I tried to use the shortcut:
"C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\Framework\DeployTools\UacElevate.exe" "C:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe" "/p:Configuration=Server /t:LaunchServerDeployWizard Deployment.btdfproj /tv:4.0"
nothing happen.
I prepared and ran a script as following (revised based on my environement):
CD
CD Windows\Microsoft.NET\Framework\v4.0.30319
MSBuild.exe "C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\Framework\BizTalkDeploymentFramework.ServerExecute.targets" /t:Deploy /p:BT_DEPLOY_MGMT_DB=True /p:ENV_SETTINGS="C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\EnvironmentSettings\SettingsFileGenerator.xml" /tv:4.0 /p:ProjectFile=Deployment.btdfproj
My deployment artifacts are preinstalled at C:\Program Files (x86)\Vancity.Isl.Core\1.2.0, got following error after ran the script:
Target Deploy:
Target ServerDeploy:
Target DeployBizTalkMgmtDB_true:
Target Bootstrap:
C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\Framework\BizTalkDeploymentFramework.targets(931,5): error MSB4062: The "GetParentPath" task could not be loaded from the assembly C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\Framework\DeploymentFramework.BuildTasks.dll. Could not load file or assembly 'file:///C:\Program Files (x86)\Vancity.Isl.Core\1.2.0\Deployment\Framework\DeploymentFramework.BuildTasks.dll' or one of its dependencies. The system cannot find the file specified. Confirm that the  declaration is correct, that the assembly and all its dependencies are available, and that the task contains a public class that implements Microsoft.Build.Framework.ITask.
Done building target "Bootstrap" in project "Deployment.btdfproj" -- FAILED.
Done building target "DeployBizTalkMgmtDB_true" in project "Deployment.btdfproj" -- FAILED.
Done building target "ServerDeploy" in project "Deployment.btdfproj" -- FAILED.
Done building target "Deploy" in project "Deployment.btdfproj" -- FAILED.
Done building project "Deployment.btdfproj" -- FAILED.
Could you please provide some more instruction how to fix it?
Thanks!

[tfabraham@4/2/2014]
The name of one of the two MSBuild tasks DLL's in v5.5 is BizTalkDeploymentFramework.Tasks.dll and in v5.0 there was only one named DeploymentFramework.BuildTasks.dll.
Are you certain that you have upgraded both the development workstations and your build server to v5.5?  Your error message is only possible with v5.0.

[xyunfan@4/3/2014]
I used the same VM to do the DEV and build right now. I uninstalled VS2010 and BTDF5.0 and installed VS2012 and BTDF5.5. but my BTDF project artifacts were generated in VS2010 & BTDF5.0, I guess that could cause some problem if BTDF 5.5 is not fully backward compatible.
Please advise,
Thanks!

[UnknownUser@4/3/2014]

[xyunfan@4/3/2014]
I found something new and had recorded in the attachment. I guess you might have a good idea how to fix it.

[xyunfan@4/3/2014]
Please confirm you received attachment.

[tfabraham@4/3/2014]
When you run the Build Server MSI command it runs the MSBuild target named "Installer".  In the Visual Studio Output window you will see the exact list of files being copied.  The expected behavior is that many files will be copied from the Deployment Framework installation (Program Files) path to the project obj\redist folder.
You should review the Output window messages in detail to find out from where BizTalkDeploymentFramework.targets is being copied twice.  Search for BizTalkDeploymentFramework.targets and check both paths.  You must still have v5.0 files around somewhere.
Another option is to just re-create your deployment project by starting again with the Add New Project wizard.  You should also try opening and building one of the sample solutions included with the Deployment Framework install, and see if you can successfully deploy and create an MSI in one of those solutions.

[xyunfan@4/3/2014]
Thanks for your advise.
I actually found the resolution to my issue. Nothing wrong with BTDF 5.5, the issue is introduced by my workaround in my code, due to our custom components, the previous BizTalkDeploymentFramework.targets generated by BTDF 5.0 didn't work well, so I have to manually modify it and override the generated BizTalkDeploymentFramework.targets in the middle of the MS build. after the migration to 5.5, I forgot to update that offline manually prepared  BizTalkDeploymentFramework.targets file in the source code, that results the issue we discussed. After I updated that file, the issue is gone.
Thanks again for all you help. I will not be able to resolve this problem without your help. BTDF is a great help to our business, and we are currently have way toward fully automate our deployment process which will save us great deal of time in near future.
Really appreciate your time, your effort.
Have a good day!

[tfabraham@4/3/2014]
Glad to hear it!  Thanks for that feedback.  It's great to hear that its saving you time and making things run more smoothly!
I'll go ahead and close this issue.
Thanks,
Tom

[UnknownUser@4/3/2014]

Issue closed by tfabraham with comment
User customizations needed to be updated for v5.5
Reason closed
Not BTDF


Add StackAid as a platform for donations

Hey @bibryam, I'm one of the founders of StackAid. Came across OSSFund on twitter recently and saw that we could submit a PR to be added the list of available platforms. Please let me know if you need any other info. Cheers.

Thanks for the addition Wes! This looks like a new promising platform! All the best.


Resolve crashes in CI for Github Actions

We are now seeing crashes in GitHub CI across all platforms and Rust versions. Tracking in https://github.com/rust-lang/rust/issues/69121

I have disabled Github Actions in the UI until this is resolved.

I wonder if this was a temporary thing which is now resolved.
i moved all my projects to github actions and they are all working well (stable, beta, nightly and specific version).

I renabled it, we shall see

Nope, still the same issue.

looks like a rustc bug worth reporting. wdyt?

Already reported above :-)

Not an expert by any means, but is there a reason for RUSTFLAGS: "-C link-dead-code"?
Does not look like that variable is set for Azure pipelines, and if I remove it, checks for nightly seem to pass for the most part.
And the other way around, running RUSTFLAGS="-C link-dead-code" cargo build --tests seem to reproduce the error locally

Nice catch! I'll look into it.

the deadcode flag is for code coverage to be more accurate


dockerfile: Split the ADD command

Currently the ADD command is IMO far too magical.  It can add local and remote files.  It will sometimes untar a file and it will sometimes not untar a file.  If a file is a tarball that you want to copy, you accidentally untar it.  If the file is a tarball in some unrecognized compressed format that you want to untar, you accidentally copy it.
There's also essentially no security for downloads.
Can we have unambiguous options, please.  For example:
COPY-IN <local file or directory> <target in image>

UNTAR <local tarball> <target directory in image>

DOWNLOAD NOVERIFY <URL> <target>

DOWNLOAD SHA256 <sha256 hash> <URL> <target>

etc.

Need the following:

To have ADD or UNPACK or EXTRACT work for at least a minimum of compressed .tar.gz and tgz compressed format. But also as many others as is possible.

Have ADD or UNPACK or EXTRACT be able to fetch and download and unpack in 1 step. From remote http:// curl/wget URL - whether it is ssl verified or not.

Once we have https://github.com/docker/docker/pull/10775 merged we can look to add options to ADD to allow for finer control over what happens.  I would prefer to do that than to add a ton of new top level Dockerfile commands that almost do the same thing as each other.  We can even then look to merge COPY and ADD, even having just those two causes confusion.

Once we have #10775 merged we can look to add options to ADD to allow for finer control over what happens

:+1:

We also need a reliable caching of ADD'ed tarballs. Docker build is supposed to set mtime from last-modified HTTP header. But isn't working for us, neither from github releases URLS (redirected to s3 with a session key). Nor from local Apache webserver.
Where #7748 is as another possible solution to our caching problem.

Hello!
We are no longer accepting patches to the Dockerfile syntax as you can read about here: https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax
Mainly:

Allowing the Builder to be implemented as a separate utility consuming the Engine's API will open the door for many possibilities, such as offering alternate syntaxes or DSL for existing languages without cluttering the Engine's codebase

Then from there, patches/features like this can be re-thought. Hope you can understand.


Fix Failing Unit Tests

Need to fix all failing unit tests so that contributors will have confidence in changes they make.

This was fixed by #26


Usage of constant addresses

Summary
Similar to https://nowarp.github.io/tools/misti/docs/detectors/ZeroAddress, but more general.
Usage of something like address("EQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAM9c") would not be detected currently, but it's as bad as the usage of zero address in some contexts.
This also should apply for newAddress function.
I don't think it makes sense to provide more context or examples, since these would be same as for ZeroAddress.

Agreed, thank you. We could generalize the zeroAddress detector creating another kind of warnings. I suggest making it optional or low severity, since it might be valid to e.g. initialize a contract with a hard-coded address.

@jubnzv Misti doesn't seem to report anything for this code:
contract Test {
    receive() {
        address("0QAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACkT"); // constant address
    }
}

@Gusarich This detector is disabled by default. You should run the analyzer with the --all-detectors CLI option. Or create a configuration file that includes the constantAddress detector.


Automatic Docker pull?

Right now, on each release, the API of the C++ and Rust languages are usually also updated, which requires the latest version of the respective docker container. In the case of C++, it's godot-cpp-compiler and for Rust godot-rust-compiler. These are created from the upstream ghcr.io/libriscv/cpp_compiler and ghcr.io/libriscv/rust_compiler.
Hence, in order to be on the latest version, one must:
docker stop godot-cpp-compiler
docker rm godot-cpp-compiler
docker pull ghcr.io/libriscv/cpp_compiler

Should this be done automatically? I don't like forcing the latest version, as people may be sitting on specific version of the extension.

Is this documented anywhere?

I still think a window for managing versions and docker running status would make sense

Also, maybe a project setting (eg cpp-api version and rust-api version) by default set to latest. People if they want can set to whatever they want in there, and then our docker pull would just get that version.
I think the default being latest wouldn't be too bad, or we could version api with sandbox(eg version 0.9 of sandbox and version 0.9 of api. Then we release api only 0.9.1, 0.9.2, ...

Fixed by https://github.com/libriscv/godot-sandbox/pull/77

Released https://github.com/libriscv/godot-sandbox/releases/tag/v0.12


TEST iohk-nix change

Opening a temporary PR top test this change https://github.com/input-output-hk/iohk-nix/pull/551 on cardano-base.
DO NOT MERGE.

They failures in the Haskell CI are probably some missing bounds to be addressed separately. Everything works in Hydra.


(GH-1704) Move CakeConsole & CakeBuildLog to Cake.Core

Make CakeConfiguration public
Fixes #1704

@patriksvensson addressed feedback.

@patriksvensson @cake-build/cake-team feedback should be addressed and rebased against latest.


Add support for an addition 'scriptService' key

Currently the Joi validator prevents additional keys being added to the fmsConnection when configuring.
This allows one to optionally add a scriptService key that can contain script name and layout to be used elsewhere. Keeping things as part of the fmsConnection config object.

This has been replaced


Sort apps per workspace in taskbar

For the taskbar, the option sort-by-workspace is the most natural sorting order of active apps. It would be nice to have this feature.

Seems possibly related to #2656, @aloispichler?


[Bug]: In List widget if there are more items then it show pagination. There is no option to make it in a single page.

Is there an existing issue for this?

[X] I have searched the existing issues

Description
Put a list widget. Add data source with 20-25 items. There will be auto pagination. There is no option to remove pagination and make it in single page.
Steps To Reproduce

Put a list widget
Add data source with 20-25 items.
There will be auto pagination
There is no option to remove pagination and make it in single page.

Public Sample App
No response
Environment
Production
Issue video log
No response
Version
v1.9.35

Hello, we're working on scroll for the List widget here #6293

Thanks for the updates Nikhil.
From: Nikhil Nandagopal @.>
Sent: Tuesday, October 17, 2023 11:16 AM
To: appsmithorg/appsmith @.>
Cc: Khambhaliya Kevalkumar Bharatbhai @.>; Author @.>
Subject: Re: [appsmithorg/appsmith] [Bug]: In List widget if there are more items then it show pagination. There is no option to make it in a single page. (Issue #28125)
"Warning: External Sender, please verify the sender email address and name before responding [or] opening attachment, URLs"
Hello, we're working on scroll for the List widget here #6293https://github.com/appsmithorg/appsmith/issues/6293

Reply to this email directly, view it on GitHubhttps://github.com/appsmithorg/appsmith/issues/28125#issuecomment-1765708887, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BDJYAV2NZBHT7BTT265HEITX7YLRJAVCNFSM6AAAAAA6CHBJRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRVG4YDQOBYG4.
You are receiving this because you authored the thread.Message ID: @.***>
[Maveric Systems] https://maveric-systems.com/
Follow us on <default.asp> [Facebook] https://www.facebook.com/mavericsystemsltd/  [LinkedIn] https://www.linkedin.com/company/maveric-systems-limited/  [Twitter] https://twitter.com/mavericsystems/  ==================================================================================================================================================================
Disclaimer: This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by return email. =================================================================================================================================================================== ============================================================================================================================================================== Disclaimer: This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by return email. ==============================================================================================================================================================


Optimise lineOut() and rename to loadBook()

Hello from this side of the internet. Nice progress.

Thanks for making a contribution! I do think that a section within the loadBook() function is still needed.
       htmlList = textwrap.wrap(htmlString,width=screenWidth)          x = 0          for i in htmlList:              fullBook.append(i)              x+=1
The way that the ebooklib loop works is that the book is split into sections of html (might be by chapter), so each time it loops, its only getting one chapter. So the function goes through a chapter, splits it in lines, then appends all of those lines to fullBook, THEN moves on to the next chapter and processes that. So I think fullBook would get overwritten with each chapter. But passing in the book location seems way better. I’ll poke around this and see what it does to the eReader.

Updated.
Ah, of course. Yeah you're right! EPUB chapters are stored in separate embedded "html" files, so your loop was correct in looping each one and appending to a separate array/list.
Future idea(s): add a loadChapter(chapterNumber) function (or similar) and a menu item to allow listing and selection of chapters (as well as jumping to a specific page - not sure how you'd "type" the digits though).
Of course all this is just "free info", do with it what you you will :)

Thanks! I’ll look at adding this in! (My first project where someone has wanted to merge a change in, its pretty exciting actually!) and I had been wrapping my head around what else I can add. While the book is open, my menu button does nothing really. It could bring up a chapter change screen. I could as a part of the loop make another list of the chapters split with what number file it is from the book. I also have some unused code for getting the book length. I just need to figure out how to get these things in the book cache file because it doesn’t know anything about the book until it gets to loadBook().


Version 2.2 not available on terraform registry.

Description
Expected behavior
Version 2.2 should be available on terraform registry
Actual behavior
it is not
Terminal Output Screenshot(s)

Additional context

Fixed


sandia openshmem PR

this PR adds support for openshmem to nim

Hello, @ct-clmsn
This PR doesn't seem to contain any commit.

@xflywind apologies - will send fix very soon.

ping

It has been almost a year, @ct-clmsn everything alright?

@PhilippMDoerner all is well, what is the block on this PR?

Really? where are your changes?

8 commits, no files changed. I haven't had this issue before. @ringabout could you check the commits tab? It shows which files have been changed and where the files have been changed. I'm not sure what the issue is on my end.

PR closed made OBE (over taken by events) by other PRs


Create a timeline

Add a list of all your activities, achievements, work experience. Add as much info as you can remember. In case you don't have working experience tell about your pet-projects or student projects. Don't be shy :)

Add skills block styles
Add experience block styles
Add skills block html
Add experience block template
Add experiences html
Fill skills block with own info
Fill experience block with own info

Divided into pet projects and jobs


Progress of Tile38 2.0

Tile38 2.0 is in the works and I wanted to share some (very early pre-alpha) benchmarks.
The big internal changes are:

R-tree data structure is entirely rewritten. It's now a customized R-star like tree with support for 2 and 3 dimensions.
All Key/value objects have a new format which take up much less memory per item.
I finally got around to adding Redcon for network IO.

I want to emphasis that this is a very early pre-alpha and I expect that the results will vary when the public preview is released. Though I do expect huge gains across the board.
 Is there a branch or build somewhere we can try?

@thisisaaronland That's great to hear. I would love any feedback you can provide.
As it stands right now Tile38 2.0 is missing a lot of components such as geofencing, replication, configuration, etc. It only has the PING,GET,SET,NEARBY commands. Maybe I can get a rough alpha branch within the next few week for you to try.

@tidwall 🏆

@tidwall Awesome! Thank you for your dedication towards improving this. Looking forward to testing the new tree structure.

Maybe I can get a rough alpha branch within the next few week for you to try.

:+1: I would also like this :smile:

Running locally on a MacBook Pro 15" 2.8 GHz Intel Core i7 using Go 1.7

The benchmarks where run on the same machine? Any idea of the impact of the new network IO on a test over network?

R-tree data structure is entirely rewritten. It's now a customized R-star like tree with support for 2 and 3 dimensions.

Is it mostly threading that has improved, or will it still be beneficial to run a multiple instances on a single core? In other words does a single Tile38 instance on a single thread also see a (major) performance increase?

@Lars-Meijer 👍 I'll let you know asap.

The benchmarks where run on the same machine? Any idea of the impact of the new network IO on a test over network?

These were run on the same machine. The base network library is Redcon, here's some benchmarks that were done a few months back. https://simongui.github.io/2016/10/24/benchmarking-go-redis-server-libraries.html

Is it mostly threading that has improved, or will it still be beneficial to run 4 tile instances on 4 cores? In other words does a single Tile38 instance on a single thread also see a (major) performance increase?

Yes, a single threaded server will see the same improvements. The underlying data structure is that much better. Also, a single threaded server will likely be even a little faster for SET than multithreaded in 2.0 because Tile38 will avoid all locking when only one CPU is being used (GOMAXPROCS=1).
Of course with --appendonly no it's even more performant.
Overall I expect SETs to be around 1.5x - 3.0x faster in 2.0 (single or multi threaded).
GETs will be around 5x - 10x faster in single threaded.

Of course with --appendonly no it's even more performant.

Will this be an official feature in 2.0?

Yes, a single threaded server will see the same improvements. The underlying data structure is that much better. Also, a single threaded server will likely be even a little faster for SET than multithreaded in 2.0 because Tile38 will avoid all locking when only one CPU is being used (GOMAXPROCS=1

Thanks for all the work, looking forward to the results and testing with the new version.

Hello, version 2.0 sounds awesome. Any update as to when it may become available?

@abelyansky 2.0 is still a work in progress. No definite date is set, but I'm hopeful for this year. And yes, it's gonna pretty awesome. :)

@tidwall - thanks for the update and the hard work involved! am excited to see the alpha version once it's ready.

So I've rolled many of the features and optimization that I planned for v2 into recent v1 releases, including  --appendonly no,  Redcon, and rtree optimizations.
I plan on continuing to improve v1 until we need breaking changes that require version 2.
I'm closing this issue for now. Thanks to everyone for all the feedback.


Issue#199

The start to the adding the l=1 odyssey.  I have added a GUI.
Still to do for just the GUI to be useful ...

[x] Add some words to the GUI (e.g., label the silders)
[x] Add [epsilon_g, coupling, d01] slider
[x] Add input from star class (either dnu/eps of mode frequencies)
[x] Check the numerical consistency of the results (being pragmatic)
[x] Add an 'output' or 'save' button.
[ ] Add a 'peakbag_mixed' class that hands everything off to pymc3.
[ ] Add epsilon changing as a linear function.
[ ] Add a button to visualise the prior we are getting from reggae.
[x] Change password to 'Ironman sucks'.

As I understood it, the gui was just a simple way for us to get a feel for what's going on in the spectrum when we slide different things. So putting in sliders for different variables and expanding the model complexity makes sense. But wouldn't it be a bit overkill to implement too much functionality apart from that? Do we really need the import pbjam and peakbagging parts for example?

What does numerical consistency mean?

What part should be modelled hierarchically?

I prefer 'everything is awesome' as a password.

@nielsenmb Good point.
I was thinking an intermediate implementation would be the sliders to do the current asy_peakbag job but for the l=1.  We could dev all the peakbag l=1 work but based on human intervention.  Then at some point we transition to replacing the sliders (Reggae) with a proper algo.  Make any sense?

Yup! That's what I was thinking.
It's a pretty nice teaching tool though, at least I got a much better sense
of what goes on today when we were looking at it. So it's definitely worth
shining it up a bit later.
On Mon, 2 Dec 2019 at 19:53, Guy Davies<EMAIL_ADDRESS>wrote:

@nielsenmb https://github.com/nielsenmb Good point.
I was thinking an intermediate implementation would be the sliders to do
the current asy_peakbag job but for the l=1. We could dev all the peakbag
l=1 work but based on human intervention. Then at some point we transition
to replacing the sliders (Reggae) with a proper algo. Make any sense?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/grd349/PBjam/pull/203?email_source=notifications&email_token=AEJWO326K3MOT6VMB6XR4DTQWVRSDA5CNFSM4JTNZ2Y2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFUWBTA#issuecomment-560554188,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEJWO366KI7UENM3AMLT72TQWVRSDANCNFSM4JTNZ2YQ
.

Ironman sucks

I have been looking at epsilon_g for our star of choice (Boeing 4448777).  Is there any theoretical justification for having an epsilon_g that isn't zero.  I'm not talking about about XYZ fitted stuff to models (screw models ;) ) but rather does epsilon_g actually mean anything?  What might epsilon_g vary with in terms of fundamental parameters?

OK - we need another meeting to start making a plan.  @nielsenmb should we chat next week?

Sure! I'm free any day expect Friday 6th.

Reggae is now integrated (not fully but getting there) with the star class and even has an implementation in plotting (plot_echelle).  I want to improve the integration a bit and fix a few of the niggles.  Then once we have docs, comments, tests, and examples in the docs we can think about merging.
If you want to try it on your favourite star (4448777 works best as some numbers are hard coded for now) then just star.run_reggae() after you have run the asy_fit.  You can even call star.reggae.plot_echelle().

I also need to add pyqt5 to the requirements.

Awesome! So it opens the UI from the star class?
Also having Reggae output a frequency list in some form is useful for some upcoming work if it doesn't already do this.

Yes - star.run_reggae() is now a thing!
It outputs some frequencies and then you can star.reggae.plot_echelle()

I'm going to close this PR as the cpnest PR has all of this stuff in there and is much more advanced.


np.array_split on pd.Series causing unavoidable deprecation warnings for Series.swapaxes

As reported in this issue on from numpy, the use of np.array_split on a pandas object results in an unavoidable
FutureWarning: 'Series.swapaxes' is deprecated and will be removed in a future version. Please use 'Series.transpose' instead.

This reply in the issue thread suggest an alternative solution to np.array_split, relying on iloc instead.

Thanks, I'll update this in the next patch release.

Resolved with v2.1.0.


Can I Use python+scrcpy to implement screenshots

I want to use scrcpy to implement screenshots and save 。because adb printscreen is 1-3s，I need faster.So Can I do it? please tell me,thank you very much.

Yes you can

I want to use scrcpy to implement screenshots and save 。because adb printscreen is 1-3s，I need faster.So Can I do it? please tell me,thank you very much.

Have you done it yet, please help share it, thanks you


Dont understand how to install

Hi I have installed gangefors airdcpp before thru docker on a synology nas, pulled the file from registry and made a container and linked the volumes working lika a charm. But cant understand how to install airdcpp-adlchecker. Can't even make a container, and the right place to make a volume. Just get
Error!="Configuration not found.
I am obviusly new to docker and tried to go through docker tutorials but im not getting much wiser. Is there a more foolproof readme?

If you are using the gangefors/airdcpp-webclient image for your application container then you should be able to use the docker command as it is stated in the README.
Just make sure that you have put a ADLSearch.xml file in the airdcpp configuration directory. If you find DCPlusPlus.xml then this is the directory to put the ADLSearch.xml file in.
Example:
$ docker run -d --name my-container \
    -e PUID=$(id -u) \
    -e PGID=$(id -g) \
    -v $(pwd)/config:/.airdcpp \
    gangefors/airdcpp-webclient
d440538315b82f9ea28bef148260b4f5b0eabbe46697f1716ec17d36502f5324

$ docker run --rm --volumes-from my-container gangefors/airdcpp-adlchecker --help
usage: adlchecker.py [-h] [--config CONFIG] [--profile PROFILE] [--no-ignore]
                     [--progress] [--no-db]

Perform an ADL search.

options:
  -h, --help         show this help message and exit
  --config CONFIG    Path to the AirDC++ config directory
  --profile PROFILE  The share profile to run the ADL against
  --no-ignore        Do not hide results from ADLSearch.ignorelist
  --progress         Print progress while scanning.
  --no-db            Don't use the FileIndex DB, instead stat files on fs.

If you still can't get the adlchecker docker image to work then you need to run the adlsearch.py python script from the configuration folder. There are some limited documentation regarding that in the script itself.
There are limitations running the script with python when running airdcpp in a container due to how share paths are mapped between the host and the container. If your DC share doesn't use the same paths in the container as on the host then the script will not be able to check file sizes correctly (since it uses the path the file have in the DC share when checking the file on the host).
This limitation can be overcome by installing plyvel, which is documented in the script itself. Then the script will be using the airdcpp file index DB directly. But if you are not familiar with python or not using linux then I think that you should keep trying to get the docker approach to work.


support for release names with a dash in it, such as 'kali-rolling'

Resolves issue #2638 .

This looks right on the money; thanks!  Could you please sign the CLA so that I can merge it?

CLA signed.

Thanks!!


Slow dataloading with big datasets issue persists

Hi,
I reported too slow data fetching when data is large(#2210) a couple of weeks ago, and @lhoestq referred me to the fix (#2122).
However, the problem seems to persist. Here is the profiled results:

Running with 60GB

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  517.96         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
model_backward                     	|  0.26144        	|100            	|  26.144         	|  5.0475         	|
model_forward                      	|  0.11123        	|100            	|  11.123         	|  2.1474         	|
get_train_batch                    	|  0.097121       	|100            	|  9.7121         	|  1.8751         	|

Running with 600GB, datasets==1.6.0

Action                             	|  Mean duration (s)	|Num calls      	|  Total time (s) 	|  Percentage %   	|
------------------------------------------------------------------------------------------------------------------------------------
Total                              	|  -              	|_              	|  4563.2         	|  100 %          	|
------------------------------------------------------------------------------------------------------------------------------------
get_train_batch                    	|  5.1279         	|100            	|  512.79         	|  11.237         	|
model_backward                     	|  4.8394         	|100            	|  483.94         	|  10.605         	|
model_forward                      	|  0.12162        	|100            	|  12.162         	|  0.26652        	|

I see that get_train_batch lags when data is large. Could this be related to different issues?
I would be happy to provide necessary information to investigate.

Hi ! Sorry to hear that. This may come from another issue then.
First can we check if this latency comes from the dataset itself ?
You can try to load your dataset and benchmark the speed of querying random examples inside it ?
import time
import numpy as np

from datasets import load_from_disk

dataset = load_from_disk(...) # or from load_dataset...

_start = time.time()
n = 100
for i in np.random.default_rng(42).integers(0, len(dataset), size=n):
    _ = dataset[i]
print(time.time() - _start)

If we see a significant speed difference between your two datasets then it would mean that there's an issue somewhere

Hi @lhoestq, here is the result. I additionally measured time to load_from_disk:

60GB

loading took:  22.618776321411133
ramdom indexing 100 times took: 0.10214924812316895

600GB

loading took:  1176.1764674186707
ramdom indexing 100 times took: 2.853600025177002

Hmm.. I double checked that it's version 1.6.0. The difference seems quite big, isn't this contrary to what you have plotted in #2122 ?

I'm surprised by the speed change. Can you give more details about your dataset ?
The speed depends on the number of batches in the arrow tables and the distribution of the lengths of the batches.
You can access the batches by doing dataset.data.to_batches() (use only for debugging) (it doesn't bring data in memory).
Also can you explain what parameters you used if you used map calls ?
Also if you have some code that reproduces the issue I'd be happy to investigate it.

Also if you could give us more info about your env like your OS, version of pyarrow and if you're using an HDD or a SSD

Here are some details of my 600GB dataset. This is a dataset AFTER the map function and once I load this dataset, I do not use map anymore in the training. Regarding the distribution of the lengths, it is almost uniform (90% is 512 tokens, and 10% is randomly shorter than that -- typical setting for language modeling).
len(batches):
492763

batches[0]:
pyarrow.RecordBatch
attention_mask: list<item: uint8>
  child 0, item: uint8
input_ids: list<item: int16>
  child 0, item: int16
special_tokens_mask: list<item: uint8>
  child 0, item: uint8
token_type_ids: list<item: uint8>
  child 0, item: uint8

Regarding the environment, I am running the code on a cloud server. Here are some info:
Ubuntu 18.04.5 LTS   # cat /etc/issue
pyarrow                 3.0.0  # pip list | grep pyarrow

The data is stored in SSD and it is mounted to the machine via Network File System.
If you could point me to some of the commands to check the details of the environment, I would be happy to provide relevant information @lhoestq !

I am not sure how I could provide you with the reproducible code, since the problem only arises when the data is big. For the moment, I would share the part that I think is relevant. Feel free to ask me for more info.
class MyModel(pytorch_lightning.LightningModule)
    def setup(self, stage):
        self.dataset = datasets.load_from_disk(path)
        self.dataset.set_format("torch")

    def train_dataloader(self):
        collate_fn = transformers.DataCollatorForLanguageModeling(
                tokenizer=transformers.ElectraTokenizerFast.from_pretrained(tok_path)
        )
        dataloader = torch.utils.DataLoader(
                self.dataset,
                batch_size=32,
                collate_fn=collate_fn,
                num_workers=8,
                pin_memory=True,
       )

Hi! I just ran the same code with different datasets (one is 60 GB and another 600 GB), and the latter runs much slower. ETA differs by 10x.

@lhoestq and @hwijeen
Despite upgrading to datasets 1.6.2, still experiencing extremely slow (2h00) loading for a 300Gb local dataset shard size 1.1Gb on local HDD (40Mb/s read speed). This corresponds almost exactly to total data divided by reading speed implying that it reads the entire dataset at each load.

Hi @lhoestq thanks for the quick turn-around, actually the plain vanilla way, without an particular knack or fashion, I tried to look in the documentation for some alternative but couldn't find any

dataset = load_from_disk(dataset_path=os.path.join(datasets_dir,dataset_dir))

I’m facing the same issue when loading a 900GB dataset (stored via save_to_disk): load_from_disk(path_to_dir) takes 1.5 hours and htop consistently shows high IO rates > 120 M/s.

I wasn't able to reproduce this on a toy dataset of around 300GB:
import datasets as ds

s = ds.load_dataset("squad", split="train")
s4000 = ds.concatenate_datasets([s] * 4000)
print(ds.utils.size_str(s4000.data.nbytes))  # '295.48 GiB'

s4000.save_to_disk("tmp/squad_4000")

import time
from datasets import load_from_disk

_start = time.time()
s4000_reloaded = load_from_disk("tmp/squad_4000")  # Blocks read 18
print(f"Elapsed time: {time.time() - _start:.02f}s")  # Elapsed time: 14.60s

Could you run this on your side and tell me if how much time it takes ? Please run this when your machine is idle so that other processes don't interfere.
I got these results on my macbook pro.

@lhoestq running as we speak, bear with me

Just tried on google colab and got ~1min for a 15GB dataset (only 200 times SQuAQ). The time is spent reading the Apache Arrow table from the memory mapped file. This might come a virtual disk management issue. I'm trying to see if I can still speed it up on colab.

@lhoestq what is your HD read speed?

@lhoestq Thank you! The issue is getting more interesting. The second script is still running, but it's definitely taking much longer than 15 seconds.

@lhoestq, @tsproilsl mmmh still writing on my side about 1h left, thinking on it are your large datasets all monoblock unsharded ? mine is 335 times 1.18Gb shards.

The 529.10s was a bit too optimistic. I cancelled the reading process once before running it completely, therefore the harddrive cache probably did its work.
Here are three consecutive runs
First run (freshly written to disk):
Blocks read 309702
Elapsed time: 1267.74s
Second run (immediately after):
Blocks read 113944
Elapsed time: 417.55s
Third run (immediately after):
Blocks read 42518
Elapsed time: 199.19s

@lhoetsq
First test elapsed time: 11219.05s
Second test running

@lhoestq
Second test

Blocks read 1265609
Elapsed time: 11216.55s

Unfortunately no. I did some tests on google colab. The first time the dataset arrow file is memory mapped takes always a lot of time (time seems linear with respect to the dataset size). Reloading the dataset is then instantaneous since the arrow file has already been memory mapped.
I also tried using the Arrow IPC file format (see #1933) instead of the current streaming format that we use but it didn't help.
Memory mapping is handled by the OS and depends on the disk you're using, so I'm not sure we can do much about it. I'll continue to investigate anyway, because I still don't know why in some cases it would go through the entire file and in other cases it would do almost no reading.

@lhoestq thanks for the effort, let's stay in touch

Just want to say that I am seeing the same issue. Dataset size if 268GB and it takes 3 hours to load load_from_disk, using dataset version 1.9.0. Filesystem underneath is Lustre

Hi, confirmed Windows issue, exact same code running on Linux total loading time about 3 minutes.

Hmm that's different from what I got. I was on Ubuntu when reporting the initial issue.

Hi,
I'm experiencing the same issue where it is taking 40 minutes (with 500GB read from disk) to load a 220GB dataset stored using save_to_disk. I'm loading from a SSD on Ubuntu 18.04, datasets 1.11.0, and pyarrow 5.0.0. After profiling load_from_disk and digging a bit into pyarrow I've found the following.

About 60% of the load time is spent in pyarrow.Table.cast used within update_metadata_with_features. Since update_metadata_with_features is only updating the schema metadata, wouldn't it be better to use pyarrow.Table.replace_schema_metadata instead of pyarrow.Table.cast? Doing this makes update_metadata_with_features almost instantaneous.
The remaining time is spent in pyarrow.ipc.RecordBatchStreamReader.read_all. Looking at Arrow C++ libraries, I see posix_madvise system calls with MADV_WILLNEED which seem to make the OS load the entire file into the page cache.

I have limited knowledge about memory mapping and page cache so I'm not sure if I'm on the right track here. But at least for me modifiying update_metadata_with_features to use pyarrow.Table.replace_schema_metadata reduces the load time to 16 mintures with 220GB read from disk. Modifying/recompiling Arrow to suppress the posix_madvise system calls further reduces the load time to under 4 mintures with 40GB read from disk. I'm not sure if there is an easy fix for the latter without manually modifing the Arrow C++ libraries.
I've also tried loading the dataset on a MacBook and while it loads faster with fewer reads (meaning part of the issue is OS-specific), the problem with update_metadata_with_features being slow remains.

Hi @arsarabi ! Thanks for investigating, these are useful insights.
Good catch about Table.cast, feel free to open a PR to use Table.replace_schema_metadata instead (plus it looks like it's not considered "experimental" anymore since pyarrow 5.0.0)
Regarding the posix_madvise system calls with MADV_WILLNEED, it's probably worth discussing about it on the Apache Arrow mailing list.

Hey all,
I've been plagued by this issue when training large language models on C4 (2.1TB Arrow file after tokenization), so wanted to spend some time doing a deep dive.
I took @lhoestq's test script, and modified it to 1000 times SQUaD so it could fit on a variety of machines I tested
import datasets as ds

s = ds.load_dataset("squad", split="train")
s4000 = ds.concatenate_datasets([s] * 1000)
print(ds.utils.size_str(s4000.data.nbytes))  # '73.90 GiB'

s4000.save_to_disk("tmp/squad_4000")

Finally, loading this dataset on my MacBook Pro yielded a similar time as @lhoestq, confirming his results:
In [3]: import psutil
   ...: import time
   ...: from datasets import load_from_disk
   ...:
   ...: disk = "disk0"  # You may have to change your disk here
   ...: iocnt1 = psutil.disk_io_counters(perdisk=True)[disk]
   ...: time1 = time.time()
   ...:
   ...: s4000_reloaded = load_from_disk("tmp/squad_4000")
   ...:
   ...: time2 = time.time()
   ...: iocnt2 = psutil.disk_io_counters(perdisk=True)[disk]
   ...:
   ...: print(f"Blocks read {iocnt2.read_count - iocnt1.read_count}")  # Blocks read 18
   ...: print(f"Elapsed time: {time2 - time1:.02f}s")  # Elapsed time: 14.60s

Blocks read 9242
Elapsed time: 4.62s

Next, I tried the same experiment on my desktop running Ubuntu 20.04 with an SSD and a 7200 RPM HDD:
Ubuntu SSD results:
In [10]: iocnt1 = psutil.disk_io_counters(perdisk=True)[disk]
    ...: time1 = time.time()
    ...:
    ...: s4000_reloaded = load_from_disk("tmp/squad_4000")
    ...:
    ...: time2 = time.time()
    ...: iocnt2 = psutil.disk_io_counters(perdisk=True)[disk]
    ...:
    ...: print(f"Blocks read {iocnt2.read_count - iocnt1.read_count}")  # Blocks read 18
    ...: print(f"Elapsed time: {time2 - time1:.02f}s")  # Elapsed time: 14.60s
Blocks read 26718
Elapsed time: 29.23s

Ubuntu HDD results:
In [6]: iocnt1 = psutil.disk_io_counters(perdisk=True)[disk]
   ...: time1 = time.time()
   ...:
   ...: s4000_reloaded = load_from_disk("tmp/squad_4000")
   ...:
   ...: time2 = time.time()
   ...: iocnt2 = psutil.disk_io_counters(perdisk=True)[disk]
   ...:
   ...: print(f"Blocks read {iocnt2.read_count - iocnt1.read_count}")  # Blocks read 18
   ...: print(f"Elapsed time: {time2 - time1:.02f}s")  # Elapsed time: 14.60s
Blocks read 24433
Elapsed time: 116.97s

I also tracked the read speed of my HDD and SSD while the file was reading using IOstat. The SSD read the Arrow file at 429 M/s, and the HDD read it at 92 M/s. This factor correlates with the difference in elapsed time.
Finally, I have viewed the block size for all platforms, and they are all 4096 bytes. Some questions that I have:

Why does the Ubuntu machine read many more blocks than the OSX machine? It isn't a virtualized disk for this experiment, so memory mapping should be doing something similar.
I thought Arrow files should have been memory mapped into the operating system, which implied to me that the time to load a dataset shouldn't correlate with the number of tokens in the dataset. Is this true? As I increase the size of the dataset (the factor that we multiply SQUaD by), the time for load_from_disk increases. This is at the point where it takes ~4 hours to initialize a job that loads a copy of C4, which is very cumbersome to experiment with.

Any thoughts would be appreciated! This is currently a blocker in my workload.

Although I hope this issue get resolved fundamentally, the following could be a workaround.

split your corpus into many small sized files, say 10GB.
create one arrow file for each small sized file
use Pytorch's ConcatDataset to load a bunch of datasets

dataset_list = [datasets.load_from_disk(p) for p in dataset_paths]
for ds in dataset_list:
    ds.set_format("torch")
dataset = torch.utils.data.ConcatDataset(dataset_list)  # use this for training

I did not measured the exact time difference, but in my case this approach made training with 600GB data feasible.

@hwijeen Ooh, thank you! Although I am curious -- if you are loading shards in serial (as opposed to parallel), shouldn't the loading time be the same as loading the entire dataset? I would understand if one was loading the dataset with multiprocessing, but wanted to check on the code snippet you gave.

It seems the loading time does not increase linearly according to the dataset size, like it is shown in the above comment.

Hi !

I thought Arrow files should have been memory mapped into the operating system, which implied to me that the time to load a dataset shouldn't correlate with the number of tokens in the dataset. Is this true? As I increase the size of the dataset (the factor that we multiply SQUaD by), the time for load_from_disk increases. This is at the point where it takes ~4 hours to initialize a job that loads a copy of C4, which is very cumbersome to experiment with.

Yes in theory it shouldn't be correlated to the size of the dataset.
Here is the function that is called to load the Arrow table of a dataset. It takes a path to an Arrow file (in streaming format) as input:
import pyarrow as pa

def _memory_mapped_arrow_table_from_file(filename: str) -> pa.Table:
    memory_mapped_stream = pa.memory_map(filename)
    opened_stream = pa.ipc.open_stream(memory_mapped_stream)
    pa_table = opened_stream.read_all()
    return pa_table

The part that can sometimes take a long time is the call to opened_stream.read_all(). Not sure what pyarrow does at that point to cause this though. In my opinion we should discuss with the Arrow team to figure out what's happening.
Although we can probably workaround this in datasets by systematically sharding/partitionning the datasets on disk if that helps.

This has been a very interesting discussion to read. Are there any updates on it? I take it that the best option we have now is to shard our data into multiple datasets and concatenate them as shown above by @hwijeen.
On a more basic note, the huggingface Dataset should play really well with the DistributedSampler object used by the Trainer  when doing e.g., DDP? I'm hoping that by using the huggingface Dataset, the data loader will just index into the pyarrow table and the dataset won't be loaded in full in each process (but we have to pay the cost of the load_data in each process presumably so that the data loader can index into the table on that process)? Then I can use the data collators and other tools available in the library to do multi-node training. Is my intuition correct?

This has been a very interesting discussion to read. Are there any updates on it? I take it that the best option we have now is to shard our data into multiple datasets and concatenate them as shown above by @hwijeen.

If this solution proves to help, we can add an arrow files sharding for all big datasets directly integrated in load_dataset.

I'm hoping that by using the huggingface Dataset, the data loader will just index into the pyarrow table and the dataset won't be loaded in full in each process (but we have to pay the cost of the load_data in each process presumably so that the data loader can index into the table on that process)?

Yes your intuition is right :)

By any chance, do we have a better understanding of what's happening?
I am encoutering a similar problem: I have an arrow file produced by HF datasets (shard+save_to_disk) and I am trying to load this dataset/arrow file with datasets.load_from_disk(the_dataset_folder). I noticed that the first time I load it, it would be significantly slower than the subsequent times. Two days later, I will retry loading it, and it will be slow again...
After diving a little bit, the gap happens in the _memory_mapped_arrow_table_from_file function, and in particular in the call to RecordBatchStreamReader.read_all:https://github.com/huggingface/datasets/blob/158917e24128afbbe0f03ce36ea8cd9f850ea853/src/datasets/table.py#L51
read_all is slow the first time (probably for some operations that are only happening once, and are cached for a few hours?), but not the subsequent times.
>>> def _memory_mapped_arrow_table_from_file(filename):
...     memory_mapped_stream = pa.memory_map(filename)
...     opened_stream = pa.ipc.open_stream(memory_mapped_stream)
...     start_time = time.time()
...     _ = opened_stream.read_all()
...     print(f"{time.time()-start_time}")
...
>>> filename_slow = "train/00248-00249/cache-3d25861de64b93b5.arrow"
>>> _memory_mapped_arrow_table_from_file(filename_slow) # First time
0.24040865898132324
>>> _memory_mapped_arrow_table_from_file(filename_slow) # subsequent times
0.0006551742553710938
>>> _memory_mapped_arrow_table_from_file(filename_slow)
0.0006804466247558594
>>> _memory_mapped_arrow_table_from_file(filename_slow)
0.0009818077087402344

My setup:

datasets version: 2.3.3.dev0
Platform: Linux-4.18.0-305.57.1.el8_4.x86_64-x86_64-with-glibc2.17
Python version: 3.8.13
PyArrow version: 9.0.0
Pandas version: 1.4.2

I realize this might be an Apache Arrow question so I ask them, but wanted to leave a message here too.

This is a good question to ask Arrow, feel free to share the thread link or the response here :)

For posterity, the answer from the apache arrow folks:

The short answer is no, you cannot remove that discrepancy.
For a memory mapped file, when data is first accessed it is brought into memory. Subsequent reads to that data doesn't require having to go back to disk, because it's already in memory. In your example, you haven't restarted your process so the file data is still in memory for the subsequent reads.
If you want more details about memory mapped files, I think this SO post seems to have some pretty good info 1.

I have the same issue still with the Oscar dataset... any solution, I was thinking about streaming the dataset but that's not an option for me now....

In release 2.6 we significantly improved the speed of iterating over contiguous data (from https://github.com/huggingface/datasets/pull/5030). Just make sure to use a for loop or load the dataset as an iterable dataset.
We also opened https://github.com/huggingface/datasets/issues/5265 to be able to switch from a map-style dataset to an iterable dataset and achieve the best throughput.

Hey, thank you. I'm using the default Trainer for MLM pre-training, I can load the dataset as IterableDataset (with the streaming option), but then I can not use the IterableDataset.take() and IterableDataset.skip() to split the dataset (I would like to have some validation data) because the dataset is divided in shards and once I run the training I got exception that take and skip methods cannot be used in that configuration.

If you have the dataset locally you can load it as an iterable dataset to get the best performance. This isn't implemented in datasets yet (see https://github.com/huggingface/datasets/issues/5265 mentioned above), but is pretty easy to do using IterableDataset.from_generator.
You just need to get shards of the dataset and choose some of them for training, and some of them for validation. Here is an example where you shard the dataset in 100 parts and choose the last one to be your validation set:
from datasets import load_dataset, IterableDataset

oscar = load_dataset("oscar", split="train")

# to get the best speed we don't shuffle the dataset before sharding, and we load shards of contiguous data
num_shards = 100
shards = [oscar.shard(num_shards=num_shards, index=index, contiguous=True) for index in range(num_shards)]

def gen_from_shards(shards):
    for shard in shards:
        for example in shard:
            yield example

train_ds = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards[:99]})
val_ds = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards[99:]})

If you have the dataset locally you can load it as an iterable dataset to get the best performance. Switching from a Dataset to an IterableDataset isn't implemented in datasets yet (see #5265 mentioned above), but is pretty easy to do using IterableDataset.from_generator.
You just need to get shards of the dataset and choose some of them for training, and some of them for validation. Here is an example where you shard the dataset in 100 parts and choose the last one to be your validation set:
from datasets import load_dataset, IterableDataset

oscar = load_dataset("oscar", split="train")

# to get the best speed we don't shuffle the dataset before sharding, and we load shards of contiguous data
num_shards = 100
shards = [oscar.shard(num_shards=num_shards, index=index, contiguous=True) for index in range(num_shards)]

def gen_from_shards(shards):
    for shard in shards:
        for example in shard:
            yield example

train_ds = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards[:99]})
val_ds = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards[99:]})

Thank you I'll try it!

Please also make sure to use the latest version of pyarrow to benefit from the best speed, or at least pyarrow 8.0.0 :)

If you have the dataset locally you can load it as an iterable dataset to get the best performance. Switching from a Dataset to an IterableDataset isn't implemented in datasets yet (see #5265 mentioned above), but is pretty easy to do using IterableDataset.from_generator.
You just need to get shards of the dataset and choose some of them for training, and some of them for validation. Here is an example where you shard the dataset in 100 parts and choose the last one to be your validation set:
from datasets import load_dataset, IterableDataset

oscar = load_dataset("oscar", split="train")

# to get the best speed we don't shuffle the dataset before sharding, and we load shards of contiguous data
num_shards = 100
shards = [oscar.shard(num_shards=num_shards, index=index, contiguous=True) for index in range(num_shards)]

def gen_from_shards(shards):
    for shard in shards:
        for example in shard:
            yield example

train_ds = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards[:99]})
val_ds = IterableDataset.from_generator(gen_from_shards, gen_kwargs={"shards": shards[99:]})

Thanks, it worked, it helped to split the dataset into shards as you suggested, but I'm curious why was the dataset so slow why is this happening bcs I do some preprocessing with the Dataset.map() function and it works quite fast (for 34GB of text) it takes around 1,5 hour to process the data (tokenization, chunk merging. etc.)  with 6 workers, but then once I'm using this preprocessed dataset the iteration is significantly slower like 10times e.g., 2.2it/s vs 5s/batch.

Cool !

but I'm curious why was the dataset so slow why is this happening bcs I do some preprocessing with the Dataset.map() function and it works quite fast (for 34GB of text) it takes around 1,5 hour to process the data (tokenization, chunk merging. etc.) with 6 workers, but then once I'm using this preprocessed dataset the iteration is significantly slower like 10times e.g., 2.2it/s vs 5s/batch.

When you process an unshuffled dataset with map, you iterate over contiguous chunks of data, which is very fast. You get the best speed when you have an iterable dataset as well, when it's based on shards of contiguous data.
This is fast because internally Arrow simply iterates over the record batches.
On the other hand, if you use a map-style dataset in PyTorch, then PyTorch samples uniformly from the files on your disk. This is slower for your disk, and also requires an extra step to get the location of the examples from an index.

takes ~1h 30min, although I already cached it.

An alternative is to load the dataset as iterable but this is not implemented yet, see #5481

Is there any way I can instantiate the (also already cached) tokenized dataset directly without having to wait until raw_datasets is instantiated first?

If you want to skip that step, next time I'd recommend you to save the dataset somewhere after tokenization (e.g. using .save_to_disk()) and reload it from there instead of relying on the cache.
Though you could look for the cached arrow files in your cache and reload the data from there if you're adventurous. You can use Dataset.from_file to reload a file, and then concatenate_datasets to concatenate all the chunks.

Thank you so much, you made my day.
The adventurous route worked out well; it now only takes ~5 min :)

@lhoestq I'm also still having this issue where load_from_disk becomes exponentially slower on larger datasets. I've tried the suggestion here to shard and then concatenate the dataset but it's still just as slow as using load_from_disk out of the box.
Do you have any other suggestions on what to try here? What is the best guidance around how to use the datasets library with >1TB datasets? I'm guessing it's not supposed to be reading the entire dataset in during a load_from_disk call.
In case it helps, this is my code for loading and concatenating.
def concatenate_ds_shards(path):
    arrow_files = get_arrow_files(path)  # get the arrow files generated by `save_to_disk(...)`
    shards = [Dataset.from_file(f) for f in arrow_files]
    return ds.concatenate_datasets(shards)

Can you try using IterableDataset.from_file instead ? This should be faster since it doesn't memory map the full dataset and doesn't read the arrow record batches metadata either.
You'll get an IterableDataset which is more suited for bigger datasets. You can find some documentation about the differences about Dataset vs IterableDataset here

@lhoestq thank you for the suggestion! Can you also use IterableDataset directly within DDP setups (multi-GPU or multi-node)? Or does some special work need to be done?

Yup using split_dataset_by_node for example (see docs here)

Can you try using IterableDataset.from_file instead ? This should be faster since it doesn't memory map the full dataset and doesn't read the arrow record batches metadata either.

@lhoestq Why memory map the full dataset is slow? It is said to be memory efficient, so I assume load_dataset should also be fast (given that only few data are loaded).
I think (if I didn't take it wrong) mmap should take nearly no time and returns immediately, and all data is read from disk later when it is actually wanted. So, no pre-fetch here. But it is not true for load_dataset, which actually reads a lot of data from disk at the very beginning.
Profiling (in this comment, and also in my profiling) shows RecordBatchStreamReader.read_all is the most time-consuming function call. So what is it reading? Is it indeed required to read these data in memory mapped scenario?

We memory map the arrow files and use RecordBatchStreamReader.read_all indeed. This does not load the data in memory so it is generally super fast.
However if you have a slow disk and your dataset has thousands of arrow files with lots of record batches then it can take a few minutes to run. This is because of RecordBatchStreamReader.read_all: it doesn't read the data but it does read the record batches metadata (e.g. their length). This is used to know the size of the full dataset for example.
For iterable datasets we don't need to know the length of the dataset in advance, so we don't need to read the metadata, which makes things faster to load.

@lhoestq Thanks for your quick reply and detailed explanation. What is the proportion of metadata accounts for all data typically?
I'd like to give more details on my experiments if you don't mind taking a look.
I'm basically calling load_datasets and then map on openwebtext, which is around 40GB (~80 cached arrow files with ~500MB each) for load_dataset and a single 17GB cached arrow file for map. Observed from iotop, both load_datasets and map read ~9GB from disk (~100MB/s for ~100 seconds each), combined ~18GB read. I believe this is abnormal: map reads 50% of its cache file.
For a better view of what's in the cache file: I specified the feature of cache file to be Features({'text': Sequence(Value("uint16"), length=2048)}) when calling map.
It's also interesting that load_datasets and map read almost the same amount of metadata. Don't know if it's a coincidence.

Thanks for investigating @RmZeta2718 , this is useful information and indeed abnormal. Not sure what would cause that but may I ask you to see if reducing the number of record batches in the mapped dataset helps ? You can try passing writer_batch_size=10_000 to do so (one record batch = 10k examples instead of the default 1k)

Sorry for the delay. @lhoestq
I found NO performance improvement after increasing writer_batch_size. But I found setting batch_size=100_000 reduces map time (and disk read) to a half (~50 seconds and ~4GB). But load_datasets doesn't have any of these options, so it remains slow.
And it's hard to test because once the data are loaded, they are cached in memory, and consecutive attempts to load the same data will return immediately.
Can you guys reproduce the problem? Testing just load_dataset is enough, I think. In my example, load_dataset read 25% of the cached data (9/40), which is also abnormal. I'm not sure what the data structure of metadata is, but I'm expecting it to be around several KB (if it is a length, then 8B per shard?).

I also faced the slow mmap when loading a large dataset (~2TB) using datasets.load_from_disk. The dataset was saved with around ~1100 shards of 2GB. I believe it seems to depend on the file system. I have access to two clusters:

In the first (smaller) cluster, the file storage servers were partitioned as ext4 and mounted as nfs4 on the compute node. datasets.load_from_disk loads the dataset in a couple of minutes. Monitoring the virtual sizeVIRT in htop, I can see that the Python process is mapping ~70 GBs of data per second.
In the second cluster, the file storage is mounted as 'lustre', and the dataset files are stored with 16 stripes (lfs getstripe). datasets.load_from_disk takes around 30–40 minutes. Monitoring the virtual sizeVIRT of the Python process in htop, I see the mapping speed is around 1-2 GBs per second.

Is there any recommendations for stripe_size or stripe-count of luster or the shards size to improve the loading speed?
I prefer not to use iterative datasets since AFAIU, it's not possible to do weighted sampling from an iterative dataset.

Can you guys reproduce the problem? Testing just load_dataset is enough, I think. In my example, load_dataset read 25% of the cached data (9/40), which is also abnormal. I'm not sure what the data structure of metadata is, but I'm expecting it to be around several KB (if it is a length, then 8B per shard?).

Reproducing these issues is not easy on our side, given they depend on the setup.
For load_datastet it would be nice to be able to control the size of the batches written on disk, feel free to open an issue if it's something you'd like to see, and we'll discuss there how to do it.

I also faced the slow mmap when loading a large dataset (~2TB) using datasets.load_from_disk. The dataset was saved with around ~1100 shards of 2GB. I believe it seems to depend on the file system. I have access to two clusters:

That's helpful information, thanks ! It seems like Lustre doesn't read at full speed with the memory mapping in datasets

Is there any recommendations for stripe_size or stripe-count of luster or the shards size to improve the loading speed?

I would try increasing the stripe size in case the memory mapping does too much unecessary readahead with the default value

Reproducing these issues is not easy on our side, given they depend on the setup.

Hey! This is what I do:
raw_datasets["train"] = load_dataset("openwebtext", split=f"train[5000:]")
raw_datasets["validation"] = load_dataset("openwebtext", split=f"train[:5000]")

I mean there is nothing special in the code, so I believe the slowing issue is general and should be able to be reproduced by a plain call to load_dataset. Environments are listed above. I'm glad to provide any other info if you need.

Reproducing these issues is not easy on our side, given they depend on the setup.

Hey! This is what I do:
raw_datasets["train"] = load_dataset("openwebtext", split=f"train[5000:]")
raw_datasets["validation"] = load_dataset("openwebtext", split=f"train[:5000]")

I mean there is nothing special in the code, so I believe the slowing issue is general and should be able to be reproduced by a plain call to load_dataset. Environments are listed above. I'm glad to provide any other info if you need.

I managed to speed up the loading time (on the Lustre file system) by mmapping the arrow shards in parallel (python preload_mmap.py see the script below) and relying on the OS to cache the mmap.
Here are some results:

Without caching (on a fresh node):

# Sequentially using datasets.load_from_disk
 Loading dataset_name: 1865.329 seconds

After calling python preload_mmap.py once to cache (the first time, it takes around 90 seconds with 16 processes).

# return only the lenght of the table from each worker
# python preload_mmap.py p # use processes and return only len of table from workers
Loading dataset_name using num of (returning len) processes=16: 42.837 seconds
# python preload_mmap.py t  # use threads and return only len of table from workers
Loading dataset_name using num of (returning len) threads=16: 105.167 seconds

# return the whole table from each worker
# python preload_mmap.py p # use processes and return tables from workers
Loading dataset_name using num of (returning table) processes=16: 367.917 seconds
# python preload_mmap.py t  table # use threads and return tables from workers
Loading dataset_name using num of (returning table) threads=16: 260.434 seconds

# Sequentially using datasets.load_from_disk (the dataset has only one split)
Loading dataset_name: 397.046 seconds

It seems that preloading the files in processes (without returning the table) speeds up subsequent load_from_disk calls. However, the communication time to return the tables for concatenation is high (I am not sure how they are pickled).
Threads are slower to mmap the table but faster to communicate. If this works on other file systems, it may be worth it to have the option to load the shards in parallel here.
# preload_mmap.py
import datasets
import os
from datasets.table import MemoryMappedTable, concat_tables
import glob
import logging
from time import perf_counter
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
import sys
import concurrent
import functools

logger = datasets.logging.get_logger(__name__)

datasets.logging.set_verbosity_info()

class catchtime:
    # context to measure loading time: https://stackoverflow.com/questions/33987060/python-context-manager-that-measures-time
    def __init__(self, debug_print="Time", logger=logger):
        self.debug_print = debug_print
        self.logger = logger

    def __enter__(self):
        self.start = perf_counter()
        return self

    def __exit__(self, type, value, traceback):
        self.time = perf_counter() - self.start
        readout = f"{self.debug_print}: {self.time:.3f} seconds"
        self.logger.info(readout)

def load_file(f, return_len=False):
    with catchtime(f"Loading {f}", logger=logger):
        ds = MemoryMappedTable.from_file(f)
    if return_len:  # process pool is slow to serialize
        return len(ds)
    return ds

def load_files(
    files, debug_name="dataset_name", num_proc=16, use_threads=False, return_len=False
):
    if use_threads:
        pool_cls = concurrent.futures.ThreadPoolExecutor
        pool_kwargs = {"max_workers": num_proc}
        debug_desc = "threads"
    else:
        pool_cls = Pool
        pool_kwargs = {"processes": num_proc}
        debug_desc = "processes"
    if return_len:
        debug_desc = "(returning table) " + debug_desc
    else:
        debug_desc = "(returning len) " + debug_desc
    with catchtime(
        f"Loading {debug_name} using num of {debug_desc}={num_proc}", logger=logger
    ):
        with pool_cls(**pool_kwargs) as pool:
            result = list(
                pool.map(functools.partial(load_file, return_len=return_len), files)
            )
    return result

def main(use_threads, return_len):
    datasets.logging.set_verbosity_info()
    logging.basicConfig(level=logging.DEBUG, format="%(message)s")
    logger.info("Starting")

    jc = "datset_name"
    local = "dataset_path"
    split = "train"
    files = glob.glob(os.path.join(local, jc, split, "*.arrow"))
    files = sorted(files)

    ds = load_files(files, jc, use_threads=use_threads, return_len=return_len)
    if not return_len:
        with catchtime(f"concat_ tables"):
            ds = concat_tables(ds)

    logger.info("done")

if __name__ == "__main__":
    use_threads = False
    return_len = True

    print(
        "Usage: \n Threads: python preload_mmap.py t \n Threads and concatenate datasets: python preload_mmap.py t c"
        "\n processes: python preload_mmap.py p \n processes and concatenate datasets: python preload_mmap.py p c "
    )
    if len(sys.argv) > 1 and sys.argv[1].startswith("t"):
        use_threads = True
    if len(sys.argv) > 2:
        return_len = False

    main(use_threads, return_len)

I managed to speed up the loading time (on the Lustre file system) by mmapping the arrow shards in parallel (python preload_mmap.py see the script below) and relying on the OS to cache the mmap.
Here are some results:

Without caching (on a fresh node):

# Sequentially using datasets.load_from_disk
 Loading dataset_name: 1865.329 seconds

After calling python preload_mmap.py once to cache (the first time, it takes around 90 seconds with 16 processes).

# return only the lenght of the table from each worker
# python preload_mmap.py p # use processes and return only len of table from workers
Loading dataset_name using num of (returning len) processes=16: 42.837 seconds
# python preload_mmap.py t  # use threads and return only len of table from workers
Loading dataset_name using num of (returning len) threads=16: 105.167 seconds

# return the whole table from each worker
# python preload_mmap.py p table # use processes and return tables from workers
Loading dataset_name using num of (returning table) processes=16: 367.917 seconds
# python preload_mmap.py t  table # use threads and return tables from workers
Loading dataset_name using num of (returning table) threads=16: 260.434 seconds

# Sequentially using datasets.load_from_disk (the dataset has only one split)
Loading dataset_name: 397.046 seconds

It seems that preloading the files in processes (without returning the table) speeds up subsequent load_from_disk calls. However, the communication time to return the tables for concatenation is high (I am not sure how they are pickled).
Threads are slower to mmap the table but faster to communicate. If this works on other file systems, it may be worth it to have the option to load the shards in parallel here.
# preload_mmap.py
import datasets
import os
from datasets.table import MemoryMappedTable, concat_tables
import glob
import logging
from time import perf_counter
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
import sys
import concurrent
import functools

logger = datasets.logging.get_logger(__name__)

datasets.logging.set_verbosity_info()

class catchtime:
    # context to measure loading time: https://stackoverflow.com/questions/33987060/python-context-manager-that-measures-time
    def __init__(self, debug_print="Time", logger=logger):
        self.debug_print = debug_print
        self.logger = logger

    def __enter__(self):
        self.start = perf_counter()
        return self

    def __exit__(self, type, value, traceback):
        self.time = perf_counter() - self.start
        readout = f"{self.debug_print}: {self.time:.3f} seconds"
        self.logger.info(readout)

def load_file(f, return_len=False):
    with catchtime(f"Loading {f}", logger=logger):
        ds = MemoryMappedTable.from_file(f)
    if return_len:  # process pool is slow to serialize
        return len(ds)
    return ds

def load_files(
    files, debug_name="dataset_name", num_proc=16, use_threads=False, return_len=False
):
    if use_threads:
        pool_cls = concurrent.futures.ThreadPoolExecutor
        pool_kwargs = {"max_workers": num_proc}
        debug_desc = "threads"
    else:
        pool_cls = Pool
        pool_kwargs = {"processes": num_proc}
        debug_desc = "processes"
    if return_len:
        debug_desc = "(returning table) " + debug_desc
    else:
        debug_desc = "(returning len) " + debug_desc
    with catchtime(
        f"Loading {debug_name} using num of {debug_desc}={num_proc}", logger=logger
    ):
        with pool_cls(**pool_kwargs) as pool:
            result = list(
                pool.map(functools.partial(load_file, return_len=return_len), files)
            )
    return result

def main(use_threads, return_len):
    datasets.logging.set_verbosity_info()
    logging.basicConfig(level=logging.DEBUG, format="%(message)s")
    logger.info("Starting")

    jc = "datset_name"
    local = "dataset_path"
    split = "train"
    files = glob.glob(os.path.join(local, jc, split, "*.arrow"))
    files = sorted(files)

    ds = load_files(files, jc, use_threads=use_threads, return_len=return_len)
    if not return_len:
        with catchtime(f"concat_ tables"):
            ds = concat_tables(ds)

    logger.info("done")

if __name__ == "__main__":
    use_threads = False
    return_len = True

    print(
        "Usage: \n Threads: python preload_mmap.py t \n Threads and concatenate datasets: python preload_mmap.py t c"
        "\n processes: python preload_mmap.py p \n processes and concatenate datasets: python preload_mmap.py p c "
    )
    if len(sys.argv) > 1 and sys.argv[1].startswith("t"):
        use_threads = True
    if len(sys.argv) > 2:
        return_len = False

    main(use_threads, return_len)

I'm very happying that I use this way to acclereate the time of the data load from 15 minutes to 45 seconds. And the size of the dataset is 7 TB.
My file system is virtio-fs. I can't its real file system because my code is in the virtual machine and i can't access the host machine. I guess it should be a distributed file system similar to Luster. I don't know why it is so slow and why multi threading can acclerate it. But what i know is it really acclerate the load time and it is vital for me. Thanks for your code.

I managed to speed up the loading time (on the Lustre file system) by mmapping the arrow shards in parallel (python preload_mmap.py see the script below) and relying on the OS to cache the mmap.
Here are some results:

Without caching (on a fresh node):

# Sequentially using datasets.load_from_disk
 Loading dataset_name: 1865.329 seconds

After calling python preload_mmap.py once to cache (the first time, it takes around 90 seconds with 16 processes).

# return only the lenght of the table from each worker
# python preload_mmap.py p # use processes and return only len of table from workers
Loading dataset_name using num of (returning len) processes=16: 42.837 seconds
# python preload_mmap.py t  # use threads and return only len of table from workers
Loading dataset_name using num of (returning len) threads=16: 105.167 seconds

# return the whole table from each worker
# python preload_mmap.py p table # use processes and return tables from workers
Loading dataset_name using num of (returning table) processes=16: 367.917 seconds
# python preload_mmap.py t  table # use threads and return tables from workers
Loading dataset_name using num of (returning table) threads=16: 260.434 seconds

# Sequentially using datasets.load_from_disk (the dataset has only one split)
Loading dataset_name: 397.046 seconds

It seems that preloading the files in processes (without returning the table) speeds up subsequent load_from_disk calls. However, the communication time to return the tables for concatenation is high (I am not sure how they are pickled).
Threads are slower to mmap the table but faster to communicate. If this works on other file systems, it may be worth it to have the option to load the shards in parallel here.
# preload_mmap.py
import datasets
import os
from datasets.table import MemoryMappedTable, concat_tables
import glob
import logging
from time import perf_counter
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
import sys
import concurrent
import functools

logger = datasets.logging.get_logger(__name__)

datasets.logging.set_verbosity_info()

class catchtime:
    # context to measure loading time: https://stackoverflow.com/questions/33987060/python-context-manager-that-measures-time
    def __init__(self, debug_print="Time", logger=logger):
        self.debug_print = debug_print
        self.logger = logger

    def __enter__(self):
        self.start = perf_counter()
        return self

    def __exit__(self, type, value, traceback):
        self.time = perf_counter() - self.start
        readout = f"{self.debug_print}: {self.time:.3f} seconds"
        self.logger.info(readout)

def load_file(f, return_len=False):
    with catchtime(f"Loading {f}", logger=logger):
        ds = MemoryMappedTable.from_file(f)
    if return_len:  # process pool is slow to serialize
        return len(ds)
    return ds

def load_files(
    files, debug_name="dataset_name", num_proc=16, use_threads=False, return_len=False
):
    if use_threads:
        pool_cls = concurrent.futures.ThreadPoolExecutor
        pool_kwargs = {"max_workers": num_proc}
        debug_desc = "threads"
    else:
        pool_cls = Pool
        pool_kwargs = {"processes": num_proc}
        debug_desc = "processes"
    if return_len:
        debug_desc = "(returning table) " + debug_desc
    else:
        debug_desc = "(returning len) " + debug_desc
    with catchtime(
        f"Loading {debug_name} using num of {debug_desc}={num_proc}", logger=logger
    ):
        with pool_cls(**pool_kwargs) as pool:
            result = list(
                pool.map(functools.partial(load_file, return_len=return_len), files)
            )
    return result

def main(use_threads, return_len):
    datasets.logging.set_verbosity_info()
    logging.basicConfig(level=logging.DEBUG, format="%(message)s")
    logger.info("Starting")

    jc = "datset_name"
    local = "dataset_path"
    split = "train"
    files = glob.glob(os.path.join(local, jc, split, "*.arrow"))
    files = sorted(files)

    ds = load_files(files, jc, use_threads=use_threads, return_len=return_len)
    if not return_len:
        with catchtime(f"concat_ tables"):
            ds = concat_tables(ds)

    logger.info("done")

if __name__ == "__main__":
    use_threads = False
    return_len = True

    print(
        "Usage: \n Threads: python preload_mmap.py t \n Threads and concatenate datasets: python preload_mmap.py t c"
        "\n processes: python preload_mmap.py p \n processes and concatenate datasets: python preload_mmap.py p c "
    )
    if len(sys.argv) > 1 and sys.argv[1].startswith("t"):
        use_threads = True
    if len(sys.argv) > 2:
        return_len = False

    main(use_threads, return_len)

I'm very happying that I use this way to acclereate the time of the data load from 15 minutes to 45 seconds. And my dataset size is on the TB scale.
My file system is virtio-fs. I can't know its real file system because my code is in the virtual machine and i can't access the host machine. I guess it should be a distributed file system similar to Luster. I don't know why it is so slow and why multi threading can acclerate it. But what i know is it really acclerates the load time and it is vital for me. Thanks for your code.


React 16 support?

Hello!
RN 0.43 require to use react-16.
It'll be great to update this lib to use React 16.
Can we do that?

Any news about this ? Someone managed to get it work with react 16 ?

At least when using latest version from master this library works perfectly with React 16 :D

Sweet ! thanks for the feedback ! I'll get to work and try it on my current config :)

Latest version is compatible with latest stable version of RN which uses React 16.


Support for ARM64v8

This project builds and runs fine on ARM64v8 (build, run logs available on request).
Is there anything that needs to be done to make this docker pullable on our platforms?
I'd be happy to submit an enabling pull-request to Docker's Official Images repo for you.
Version tested: 8.4

@lag-linaro : I am also working on Haskell for arm64v8.
But in my case, I can observe that there are build time failures dues to below missing dependencies.
E: Unable to locate package ghc-8.4.3
E: Couldn't find any package by glob 'ghc-8.4.3'
E: Couldn't find any package by regex 'ghc-8.4.3'
E: Unable to locate package cabal-install-2.2
E: Couldn't find any package by glob 'cabal-install-2.2'
E: Couldn't find any package by regex 'cabal-install-2.2'

Can you please recheck whether it is really building without any modification or have you have modified the versions in your Dockerfile.

We have a policy of supporting only the two most recent minor versions of GHC.  When I looked into non-x86 architectures last (see: https://github.com/freebroccolo/docker-haskell/issues/62 ) I found that support was limited and deemed "community".  Based on comments above, it looks like the haskell.org packages we depend on are outdated for aarch64.  Ideally these packages would stay up to date with GHC releases and we could consume them -- someone want to track this down? It looks like aarch64 binary builds are also missing from upstream downloads.  I'd like to avoid building from source unless we've exhausted both of these options.

@psftw - Exactly, This is what I trying to ask.
We need new builds for GHC & Cabal for arm64v8 or aarch64 architecture.

One option is to use the approach being followed by @lag-linaro .
This eliminates the concern to pick stable and latest supported version for ghc  and cabal.
In addition, this approach is always going to work.
And once we have newer versions of ghc and cabal at Debian:Stretch or at Haskell-downloads for aarch64, then it can be picked automatically by Docker at runtime and keep on working.
I would give :+1:  to @lag-linaro approach.

Hi @lag-linaro ,
Can you go ahead with PR of your changes?
In my view changes made by you should go ahead but with a caution that we should create new folder for arm64v8 as we will be supporting older versions of packages as compared to new packages in arm64.

@odidev, please bear in mind that the patch above was only ever designed for testing purposes.  As already expressed by @psftw, the aforementioned approach is not in line with the current policies of this project.  I can currently attempting to lobby the ghc guys to re-enable support for ARM64v8.  If successful ghc releases should be in parallel with X86 and the issues we're experiencing will just go away.
Please leave this with me.  I will update when I have more information.

Thanks, @lag-linaro .
Just for more clarity, I have managed to compile and run Cabal-2.2 & 2.5 at my end.
So right now, the only dependency is on GHC.
Regards,

The dependency is on having official ghc and cabal packages published for ARM64v8.
Hopefully we can gain some traction on this soon.

Quick update: Still working on this.
I am in conversation with the GHC guys to re-enable AArch64 support.

Hi,
Is there any update regarding the progress of this activity?
Eagerly waiting for GHC for AARCH64
Regards,

Discussions with Ben ran dry, even after putting him in contact with our Toolchain guys.

Apparently Ben has been working on AArch64 recently.
Fingers crossed!

We have a policy of supporting only the two most recent minor versions of GHC. When I looked into non-x86 architectures last (see: #62 ) I found that support was limited and deemed "community". Based on comments above, it looks like the haskell.org packages we depend on are outdated for aarch64. Ideally these packages would stay up to date with GHC releases and we could consume them -- someone want to track this down? It looks like aarch64 binary builds are also missing from upstream downloads. I'd like to avoid building from source unless we've exhausted both of these options.

FYI, GHC AArch64 support will be released as part of 8.8.1, which is due imminently. \o/

@psftw the newly released 8.8.1 now supports aarch64.
Could we please add support do the Docker Official Images as well please?  \o/

Interesting info.
How can we get to know whether this is now relevant for aarch64 too?
deb http://downloads.haskell.org/debian stretch main
I am facing issues while fetching GHC-8.8.1 via above repo.

Support is currently lacking.  I have reported this.
https://downloads.haskell.org/debian/dists/stretch/main/

Ideally we can use upstream Debian packages, but in the meantime we could build an aarch64 image based on the binary release. I've seen reports of that having issues as well, but have not played with it yet.

Today I played with GHC on aarch64 (via AWS a1). I was able to get it working using the binary release (Debian packages not available) with some tweaks to the Dockerfile. I was not able to get cabal-install working -- it complains about not being able to find LLVM no matter what I've tried so far. Stack appears to work.
$ docker run -it --rm haskell
GHCi, version 8.8.1: https://www.haskell.org/ghc/  :? for help
Prelude> System.Info.arch
"aarch64"

/cabal-install-<IP_ADDRESS># ./bootstrap.sh
Using gcc for C compiler. If this is not what you want, set CC.
Using /usr/lib/gcc/aarch64-linux-gnu/6/collect2 instead.
Checking installed packages for ghc-8.8.1...
deepseq is already installed and the version is ok.
binary is already installed and the version is ok.
time is already installed and the version is ok.
transformers is already installed and the version is ok.
mtl is already installed and the version is ok.
text is already installed and the version is ok.
parsec is already installed and the version is ok.
network-uri-<IP_ADDRESS> will be installed from local directory.
network-<IP_ADDRESS> will be downloaded and installed.
HTTP-4000.3.12 will be downloaded and installed.
zlib-0.6.2 will be downloaded and installed.
random-1.1 will be downloaded and installed.
stm is already installed and the version is ok.
hashable-<IP_ADDRESS> will be downloaded and installed.
async-2.2.1 will be downloaded and installed.
base16-bytestring-<IP_ADDRESS> will be downloaded and installed.
base64-bytestring-<IP_ADDRESS> will be downloaded and installed.
cryptohash-sha256-<IP_ADDRESS> will be downloaded and installed.
resolv-<IP_ADDRESS> will be downloaded and installed.
mintty-0.1.2 will be downloaded and installed.
echo-0.1.3 will be downloaded and installed.
edit-distance-<IP_ADDRESS> will be downloaded and installed.
ed25519-<IP_ADDRESS> will be downloaded and installed.
tar-<IP_ADDRESS> will be downloaded and installed.
digest-<IP_ADDRESS> will be downloaded and installed.
zip-archive-0.3.3 will be downloaded and installed.
hackage-security-<IP_ADDRESS> will be downloaded and installed.
Cabal is already installed and the version is ok.

Using local directory for network-uri-<IP_ADDRESS>.
[1 of 1] Compiling Main             ( Setup.hs, Setup.o )

<no location info>: error:
    Warning: Couldn't figure out LLVM version!
             Make sure you have installed LLVM 7
ghc: could not execute: opt

Error during cabal-install bootstrap:
Compiling the Setup script failed.

The Dockerfile (minus cabal stuff) in case others wanted to play with ideas:
## Dockerfile for a haskell environment
FROM debian:stretch

## ensure locale is set during build
ENV LANG            C.UTF-8

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        dirmngr \
        g++ \
        git \
        gnupg \
        libnuma1 \
        libsqlite3-dev \
        libtinfo-dev \
        make \
        netbase \
        openssh-client \
        xz-utils \
        zlib1g-dev && \
    curl -fSL https://downloads.haskell.org/~ghc/8.8.1/ghc-8.8.1-aarch64-deb9-linux.tar.xz -o ghc.tar.xz && \
    curl -fSL https://downloads.haskell.org/~ghc/8.8.1/ghc-8.8.1-aarch64-deb9-linux.tar.xz.sig -o ghc.tar.xz.sig && \
    gpg --batch --keyserver ha.pool.sks-keyservers.net --recv-keys FFEB7CE81E16A36B3E2DED6F2DE04D4E97DB64AD && \
    gpg --batch --verify ghc.tar.xz.sig ghc.tar.xz && \
    mkdir /ghc-install && \
    tar xf ghc.tar.xz -C /ghc-install --strip-components=1 && cd /ghc-install && \
    ./configure --prefix /opt/ghc/8.8.1 && \
    make install && cd && \
    curl -fSL https://github.com/commercialhaskell/stack/releases/download/v2.1.3/stack-2.1.3-linux-aarch64.tar.gz -o stack.tar.gz && \
    curl -fSL https://github.com/commercialhaskell/stack/releases/download/v2.1.3/stack-2.1.3-linux-aarch64.tar.gz.asc -o stack.tar.gz.asc && \
    export GNUPGHOME="$(mktemp -d)" && \
    gpg --batch --keyserver ha.pool.sks-keyservers.net --recv-keys C5705533DA4F78D8664B5DC0575159689BEFB442 && \
    gpg --batch --keyserver ha.pool.sks-keyservers.net --recv-keys 2C6A674E85EE3FB896AFC9B965101FF31C5C154D && \
    gpg --batch --trusted-key 0x575159689BEFB442 --verify stack.tar.gz.asc stack.tar.gz && \
    tar -xf stack.tar.gz -C /usr/local/bin --strip-components=1 && \
    /usr/local/bin/stack config set system-ghc --global true && \
    /usr/local/bin/stack config set install-ghc --global false && \
    rm -rf "$GNUPGHOME" /var/lib/apt/lists/* stack.tar.gz.asc stack.tar.gz ghc-install ghc.tar.xz ghc.tar.gz.sig

ENV PATH /root/.cabal/bin:/root/.local/bin:/opt/cabal/3.0/bin:/opt/ghc/8.8.1/bin:$PATH

## run ghci by default unless a command is specified
CMD ["ghci"]

Also noticed upstream is working through some issues with aarch64: https://gitlab.haskell.org/ghc/ghc/issues/16796 and it is also still "tier 2", per https://gitlab.haskell.org/ghc/ghc/wikis/platforms so for now this may continue to be stalled.


Question : How to generate recovery codes for TOTP MFA use case in V3?

Please don't use issues for asking questions
One way to generate them is to use generate 40 or 80 random bits and base32 encode it


Changing EKF2 parameters for handheld testings

I assume that the estimated kalman filter synthesizes sensor measurements with a predictive model of how the quadrotor should behave. Is there a way to tune the filter such that it relies exclusively on sensor data? This would be helpful for handheld testing when the movement of the drone doesn't correspond at all to commands from the RC transmitter.

The EKF does not use a dynamic model of the vehicle for estimation.


The color picker is always hiding when mouse is clicked outside of the component

Current behavior: When mouse is clicked outside of the opened color picker it's isOpen value is changed and it's being closed.
Expected behavior: There is a flag for changing the default behavior, so the color picker is closed:

When button is clicked OR when the mouse down button is clicked outside of the component
ONLY when the button is clicked

It will make the picker more customizable.

I have the code proposition ready, but can't push to the repo (Access denied)
If you could help me out with that, I am glad to make a PR.

Thanks! I published a new release. I renamed the props disableCloseClickOutside!


JU-885: tech debt for confirmation statement api

[x] - remove hc path from listener path
[x] - updating terraform-modules version + adding filesystem flag set to false
[x] - readme (already updated)
[x] - human_log removed from ecs-service-config-dev

CI: IaC Security warnings found!

CI: No IaC security warnings found


Add Spring Data's @Id

@aureamunoz if you have some spare time, it would be great if you can merge this and do a 2.1.SP2 release.
Thanks!


issues loading module in Raijin

Hi there,
Have the following messages when trying to install the api in Raijin. Any ideas?
$ module use /projects/el8/opt/modules/modulefiles
$ module load agdc-api/0.1.0-b20150513
mpi4py/1.3.1 is only available for python/2.7.3 and python/2.7.5
mpi4py/1.3.1(3):ERROR:151: Module 'mpi4py/1.3.1' depends on one of the module(s) 'python/2.7.5'
mpi4py/1.3.1(3):ERROR:102: Tcl command execution failed: source /opt/Modules/extensions/extensions.tcl

agdc-api/0.1.0-b20150513(10):ERROR:105: Unable to locate a modulefile for 'psutil'
agdc-api/0.1.0-b20150513(10):ERROR:105: Unable to locate a modulefile for 'numpy'
agdc-api/0.1.0-b20150513(10):ERROR:105: Unable to locate a modulefile for 'scipy'
$

Sorry for the late reply.
The first warnings are related to dependencies.  So you basically need to have those modules loaded first.  The module load system has become somewhat unwieldly with many conflicts as the modules would try and load the "default" everything for you.  This can cause conflicts if a user already has a different module loaded.  So for the time being we've used a dependency check, to warn, and specify the exact versions that are required.
The second set of errors indicate that the module system can't find a specific module, eg.numpy.
I've just had a look at the modules under el8, and there's not much there.  I'm unsure as to who was maintaining it.  I use project u46 for loading my modules, which I can make a guess that you're not a member of.
You could try replacing el8 with u46 in your module use line, but I'm probably a little ambitious on it working for you.
In the meantime I can try and copy the modules across, and change the project access to el8.

Thanks Josh, I haven't had a chance yet, but I'll try what you suggest tomorrow.
Cheers
Juan


Section 3: First service: twitter-to-kafka-service

Section 3: First service: twitter-to-kafka-service - How to use Twitter4j & Kafka producers