use connect() with immutable js

I create redux store use the facebook immutable js.
so I get state in my component like this:
@connect(state => { rules : state.get('rules').toJS() })
because connect only receive the plain object, so I need use toJS to pass object to connect.
there is a problem , my rules is a array of object like I.fromJS([{id: 1} , {id: 2}, {id, 3}])
so every time state.get('rules').toJS() result is not equal. so when store change, this component always will rerender, because the shouldComponentUpdate in connect() only make a shallow equal. not a deep equal.
so how can I use the Immutable.is for shouldComponentUpdate in connect(), or other method to improve the performance with the array of object props equal problem?

toJS() negates many benefits of Immutable because you’re spending work assembling plain objects.
While technically you can ensure referential identity with Reselect because selectors memoize their outputs when the inputs are the same, I would suggest just using Immutable props in your components directly. Records make this fairly neat, with no need for get() accessors for properties.

@gaearon Hi! Sorry for posting on such an old post (I didn't want to create a new issue that is essentially the same).
The redux docs suggest using toJS HOC on mapStateToProps (https://redux.js.org/docs/recipes/UsingImmutableJS.html#use-a-higher-order-component-to-convert-your-smart-components-immutablejs-props-to-your-dumb-components-javascript-props) while you advice against doing so.
So my take is this: I'm using reselect's createStructuredSelector alongside immutablejs for my props. This should make immutable.js only change its parameters when they actually change, and reselect should memoize the results of those parameters so it'd only trigger when changes happened.
The point is: the redux best practices state that I should NOT use immutable objects in my dumb components. I have 2 folders on my project: components (dumb components that are for presentation only) and containers (smart components connected to the store), so basically, my components folder should work with plain JS objects, while my smart components should deal with immutable objects and pass over plain objects to the dumb ones.
So I'm thinking that implementing the HOC the redux docs state is a smart step forward, but I really wanted your input on the matter. Is this overkill? Should I just deal with immutable everywhere (I really don't feel like I'm sacrificing that much maintainability, portability or easier testing by doing so, but I may be wrong).
Thanks for your time!

The redux docs suggest using a toJS HOC on mapStateToProps

No, they don't. In the HOC example there is no toJS in mapStateToProps.
Moreover, the same page explicitly says in big letters "Never use toJS() in mapStateToProps"

@apaatsio sorry, not sure what happened there. I probably misread or got confused. Thanks for clarifying that out!

@apaatsio , so we should use toJS() in body of component, right?. If not use it, how to get value from immutable?

@apaatsio , so we should use toJS() in body of component, right?. If not use it, how to get value from immutable?


Incorrect documentation of load_kernel() spice function on the TudatPy API website

I was reading through some of the spice functions looking for load_standard_kernels(). I read the documentation for load_kernel(), which states that the function cleares all loaded kernels, but it should load a specific kernel, I imagine. The documentation is identical to the clear_kernels() function, where the description does fit the function name.

Hi Sean, thanks for pointing this out! @gaffarelj, next time you make an update of the API docs, could you work this fix in as well?

Turns out the docstring was correct in tudat-multidoc, but the clear_kernels() doctrings was incorrectly loaded for the load_kernel() function. Good catch.
I just pushed a fix to this (b5cde5d). We will however have to wait for a little bit before seeing the changes on the API docs website (a new tudat conda package has to be built first to incorporate a new little addition).

I will leave this issue opened for now, to close it only when I am certain that this is fixed on the online API website.

I can confirm this is fixed, see here.


Relaunch main exe after update

Tested briefly in my own project, but it worked as expected.
Also made the console exe hide itself upon launch.
Issue: #9

Great! thanks for your contribution, I will merge as soon as possible


Indexing a dataframe with a 2D boolean array?

It's not clear to me from the docs what exactly is meant by "boolean array". It seems from all of the examples I've seen that .loc expects a one dimensional array that just selects rows or columns (e.g. https://github.com/pydata/pandas/issues/5257).
Should it be possible to select using a 2D array (the same shape as the dataframe)? This is possible in R, and returns a vector of the values selected, although it would be nice to also have the option of returning a dataframe with NaNs where the False values are (this is very useful for un-gap-filling data, if you have an index of what was gap filled).
e.g.:
In [2]: a = pd.DataFrame(dict(a=[1,2,3], b=[4,5,6]))

In [3]: a
Out[3]:
   a  b
0  1  4
1  2  5
2  3  6

In [4]: (a%2==0).as_matrix()
Out[4]:
array([[False,  True],
       [ True, False],
       [False,  True]], dtype=bool)

# This doesn't work, it gives ValueError: too many values to unpack (expected 1)
In [5]: a.loc[(a%2==0).as_matrix()]
# But this result would be nice:
Out[6]:
   a  b
0  NaN  4
1  2    NaN
2  NaN  6

Maybe with a collapse= option to loc, to return a series instead of a DF. Not sure how that would work with multiple data types though...
Or perhaps there is already a way to do something like this?

Ah, .mask() (and .where(), inverted..). I guess this is not necessary then.


(NoRegisteredProviderFound) No registered resource provider found

AZ CLI version: 2.35.0
Environment:
React app + MSAL, SWA w/ AD authentication, byo function connected to API, with AD App registration.  Azure DevOps, via YML and BICEP.  In EastUS2.
BICEP to create:
resource staticWebApp 'Microsoft.Web/staticSites@2020-12-01' = {
name: '${longPrefixName}webapp'
location: paramLocation
sku: {
name: 'standard'
tier: 'standard'
}
properties: {
buildProperties: {
  apiLocation: ''
  skipGithubActionWorkflowGeneration: true
}

}
dependsOn: [
functionApp
]
}
Bug: attempts to link or get information about the link relationship with the BYO functions produce various AZ CLI faults, that say:
(NoRegisteredProviderFound) No registered resource provider found for location 'eastus2' and API version '2021-03-01' for type 'staticSites/userProvidedFunctionApps'. The supported api-versions are '2020-12-01'. The supported locations are 'westus2, centralus, eastus2, westeurope, eastasia'.
Code: NoRegisteredProviderFound
Message: No registered resource provider found for location 'eastus2' and API version '2021-03-01' for type 'staticSites/userProvidedFunctionApps'. The supported api-versions are '2020-12-01'. The supported locations are 'westus2, centralus, eastus2, westeurope, eastasia'.
To Reproduce

"az staticwebapp functions show" produces this error
"az staticwebapp functions link" produces this error

Linking via the portal interface works.
Additional context
Add any other context about the problem here.

Hi, I am sorry to hear you are having issues with the Azure CLI. This issue is not related to the Static Web Apps CLI.
The Azure CLI lives in this other repo: https://github.com/Azure/azure-cli.
I am going to close this issue.
Thank you.


Explore using react-navigation native-stack on web as well

Problem
To achieve 100% cross-platform code and user exeprience, we thrive to use the same code and technology on all platforms. Not all technology is ready for such cross-platform use or it simply does not want to be designed that way. However, we should look into way to limit a code platform-fragmentation as much as possible as its improving the code quality and readability too.
In this case, we are looking into using the native-stack in native platforms to gain from the native feel and potential performance boost. That leaves us with the react-navigation/stack for web.
Solution
Let's explore what is missing in the native-stack, that App might need in its web implementation.
Then once we have better idea of what effort it would need to make native-stack the only stack we use in the App, we can discuss with the maintainers, if its something they are interested in and would be supportive of. If yes, let's do it and unify the code base.

I can work on this ticket 🙌

Introduction 👋
In this post I'm trying to describe:

what we need to have in native-stack to be able to use it on web (to have cross platform code for navigation);
potential ways for achieving this

⁉ 🔑 Key aspects preventing native-stack from being used on web
Here is a preliminary report of what needs to be implemented in native-stack before we can start to use it on web 👀
⚠ Missing props from specification
animation
Right now you can specify any animations and transitions will not be animated, because code in native-stack is not taking this property into consideration.
I think it was made with a purpose, because otherwise you'll have to replicate a lot of functionality/animations/transitions on web, but the idea behind native-stack navigator was to re-use native navigation primitives.
Anyway, this is a critical point, since without animated transitions application on web loses all attractiveness 😔
animationTypeForReplace
I think it is not implemented right now because there is no animations at all on web platform.
But we are using this property, so it should be implemented along with animation property.
⛔ Missing navigation events
All events on web are not getting dispatched (I think mainly because of the fact, that all transitions are not animated, so you can use plain useEffect to detect a moment when screen is fully visible).
The code in Expensify is heavily relies on transition events, such as transitionEnd etc. and without these events app is not working at all (infinite loading Search and other screens)
📚 Properties that we rely on
cardStyle
We are actively using this property and do some modification of this property depends on screen size. Right now without this property our app is totally broken on web, because a list of chat can not be displayed because parent size is null, and we never mount our Sidebar component, and because of that messages can not be loaded.
So this is a very critical property for Expensify app to work properly 😅
🎣 Hooks
For Overlay component we are using useCardAnimation to get animation progress and then do a custom interpolation (changing opacity property).
📝 Proposed design
Below I have two proposals in my mind that we can follow for implementation.
1⃣ Expose navigation primitives from react-native-screens
Taking all design concepts into consideration I think the best way to implement this feature would be to expose "native" primitives from react-native-screens on web.
I understand, that there is no navigation primitives on web, but I think what we can do is to create an API that will have an identical API of what we have at the moment in react-native-screens and try to replicate such behavior on web (i. e. implement all animation variations, expose onAppear and other callbacks).
2⃣ Replicate all logic in native-stack
The second approach (which I think is less desirable because it'll break design principles) is to implement all these transitions on native-stack level (thus we'll not add a new code to core react-native-screens package and instead of that polyfilled functionality will be fully in native-stack package).
PoC implementation
I did all my experiments in https://github.com/Expensify/App/commit/f59566739f42bf2aa868b0f4bcd8bff64a533562
It's not fully working application, but it contains changes that allows to run app at least.

@mountiny whom we should ask to review it? 👀

Thank you @kirillzyusko I think that is a great summary. We can ask the react-navigation maintainers if they would consider these changes

@mountiny yep, that would be the next step 👀

I have asked SWM for now

ok, thanks for looking into this but given that web support for native-stack is not a priority for react-navigation, I think we should close this now.

I can see that it has been closed but I think that both of the proposed solutions would end up being a copy of the stack implementation since we would not want to invent the wheel. Or am I missing some context @kirillzyusko ?

If it were my project, I'd happily extract common logic between stack and native-stack to be reused on web where appropriate, to provide better feature parity across platforms in native-stack. But if we just have to shim to use stack on web and native-stack on mobile, that's fine for now


Support 2FA authentication

If a user does enable the 2-factor authentication option in the web app, when logging in through the browser, they are asked for a 6-digit code in addition to their password.
In the mobile app, we do not support that.  A user is let in after only entering the password, completely ignoring the 2nd authentication.

Note that 2FA is not actually available in production with Zulip right now (the feature is unfinished there).  That said, part of the reason we haven't made it a documented feature is that we haven't done the mobile work for it.

Good to know. I was concerned that we might have shipped a feature (or be about to do so) that we haven't implemented support for.


derived_from link type is missing

Describe the bug
The STAC spec includes a derived_from relation type: https://github.com/radiantearth/stac-spec/blob/master/item-spec/item-spec.md#relation-types
We don't have it. We should.

but at least between this and #66 the results are hilarious


[DOCS] or [BUG]: How does test detection work when using typescript and neovim-vim-test // how to debug

Hi friends, I use neotest native testing adapters for a few languages (rust, ruby, elixir, python) but delegate to neotest-vim-test for all others, in particular Typescript because ostensibly vim-test can integrate with the AVA framework which I use instead of jest.
However I think my configuration must be wrong and I would like to debug. It may be workspace related however. Here are my symptoms:
in a typescript or javascript file, when I call my binding to require("neotest").run.run() I get No tests found.
in a typescript or javascript file, When I call vim-test's native :TestNearest I am able to run my tests.
in a Go file, both vim-test and neotest work natively as expected, neotest invoking vim-test
in a Rust file, both vim-test and neotest work natively as expected, but neotest doesn't call vim-test
My neovim-test config (lazy.nvim) is like that:
			require("neotest").setup({
				status = {
					signs = false,
					virtual_text = true,
				},
				adapters = {
					require("neotest-rust"),
					require("neotest-python"),
					require("neotest-elixir"),
					require("neotest-rspec"),
					-- vim-test for all other types. use exclusion to match above lines.
					require("neotest-vim-test")({ ignore_filetypes = { "python", "rust", "ruby", "elixir" } }),
				},
			})

I would love any guidance you have as to how I can figure out where the ball is being dropped. It may be a special behavior given the various eccentricities of javascript ecosystem. But I would like to know how to follow my nose towards why no tests are detected in this case.

As I continue debugging this, I have discovered these log lines:
INFO | 2023-08-14T20:15:41Z-0600 | ...l/share/nvim/lazy/neotest/lua/neotest/lib/subprocess.lua:84 | CHILD | Connected to parent instance
INFO | 2023-08-14T20:15:41Z-0600 | ...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:380 | Initialising client
INFO | 2023-08-14T20:15:41Z-0600 | ...are/nvim/lazy/neotest/lua/neotest/client/events/init.lua:48 | Emitting starting event
DEBUG | 2023-08-14T20:15:41Z-0600 | ...are/nvim/lazy/neotest/lua/neotest/client/events/init.lua:50 | Calling listener summary for event starting
INFO | 2023-08-14T20:15:41Z-0600 | ...al/share/nvim/lazy/neotest/lua/neotest/adapters/init.lua:18 | Found 1 adapters for directory /Users/eve/[redacted]
DEBUG | 2023-08-14T20:15:41Z-0600 | ...al/share/nvim/lazy/neotest/lua/neotest/adapters/init.lua:19 | Adapters: { {
    adapter = {
      build_spec = <function 1>,
      discover_positions = <function 2>,
      filter_dir = <function 3>,
      is_test_file = <function 4>,
      name = "neotest-rspec",
      results = <function 5>,
      root = <function 6>,
      <metatable> = {
        __call = <function 7>
      }
    },
    root = "/Users/eve/[redacted]"
  } }
INFO | 2023-08-14T20:15:41Z-0600 | ...ocal/share/nvim/lazy/neotest/lua/neotest/client/init.lua:281 | Searching /Users/eve/[redacted] for test files

From this, I can see that for some reason neotest thinks this is a Ruby test suite, when in reality they are all Typescript files. I am unsure at this time on what basis it makes this determination but i bet neotest-rspec doesn't pick up any .test.ts files, which is why it says No tests found.

It's because neotest-rspec treats any directory with a .gitignore as a root directory and so neotest-vim-test is not used. You'll need to open an issue there to change this behaviour.
Ref: https://github.com/olimorris/neotest-rspec/blob/5fe7d860def0539f7f5d375fbf9c481c097062c8/lua/neotest-rspec/config.lua#L11-L13


Align naming conventions with Frictionless standards

Currently there are some inconsistencies with the names. Needs to be aligned
Entities:

[ ] Schema - main descriptor for the "entity", contains all the information about the package
[ ] Subschema - descriptor for "sub-entities" (e.g. media or observation for camtrap)
[ ] DataSchemaField - descriptor for properties (e.g. locationName in deployments)

In Frictionless those entities are:

Schema -> Data Package (typically referred to as package in Frictionless software): All information about a dataset, i.e. it's metadata and its resources
Subschema -> either:

Data Resource (resources in json): All information about a file (or collection of files with the same format), including path(s), format, encoding and (for tabular data) schema
Table Schema (schema in json): Description of fields in tabular data as well as missing value representation, primary keys and foreign key relationships with other resources.

DataSchemaField -> An element in the array fields of a Table Schema

Thanks Peter.
We call the result as "Data Package", whereas this "Schema" is only a descriptor of the future result. We need something not to confuse those two.

@peterdesmet Do you think this will work?

DataPackageSchema
DataPackageTableSchema
DataPackageField

This will affect a lot of things and also break current data package resources. I'll re-create them on IPT3

Also we need to decide on the camtrap data package identifier, currently http://rs.gbif.org/schemas/camtrap-dp. We can still change it

Decided on the following name conventions:
Directory structure and file names:
rs.gbif.org/
  data-packages/
    index.json # previously schemas.json
    camtrap-dp/
      1.2/
        profile/
          camtrap-dp-profile.json
        table-schemas/
          deployments.json # moved down
          media.json # moved down
          observations.json # moved down
        version.json # moved up
        index.json # previously _schema/schema.json

JSON structure for a rs.gbif.org/data-packages/<name>/<version>/index.json:
{
    "identifier"
    "url"
    "profile" # previously metadataProfile
    "name"
    "version"
    "tableSchemas": [] # previously subSchemas
      "tableSchemasRequirements": {} # previously subSchemaRequirements
}

JSON structure for rs.gbif.org/data-packages/index.json:
{
  "dataPackages": [ # Optionally, remove this property and start array at root
    {
      "identifier" # etc, same as above
    }
  ]
}


Make sure that the document is rendered on zooming and rotation for PDFViewer instances using the defaultRenderingQueue

In viewer.js we have code that ensures that the document is re-rendered on zooming and rotation. However, for components based viewers this might not work correctly, since there's currently no code which handles that.
Note that there is a good chance that this "just works" in many components viewers already, thanks to the watchScroll function.
The explanation is that zooming or rotation, most of the time, causes the viewer to change its (scrollable) size, thus triggering PDFViewer_update through PDFViewer_scrollUpdate.
However, in general there's no guarantee that this will actually work (since zooming and rotation doesn't necessarily change the size of the viewer for all documents), and requiring every viewer components implementer to provide methods for this doesn't seem like a great idea.

/botio-window preview

/botio-windows preview

@timvandermeij Would you be up for reviewing this patch?
To test this, compare the patch with the current master (remember to do node make generic components after applying the patch), by visiting the simpleviewer example locally. Then open the console (Ctrl+Shift+K in Firefox), and execute e.g. the following commands:

pdfViewer.currentScaleValue = 1;
pdfViewer.pagesRotation = 90;

/botio-linux preview

Looks good, thank you for the patch and for providing the clear testing steps!


Color

Hi, how to change color with minivim?
Cause colors aren't well done.
I can't see comment in my file cause they're in dark color.

Hi,
"Colors" are in fact called "colorschemes" in Vim.
And yes, I bundle a theme with Minivim. To change them you can :

Delete the lines from """" Color Scheme to the end in the MiniVim source file to revert to the vanilla Vim
Change the colorscheme at run time by using :colorscheme default by example

Let me know if this works for you ;-)

Unless you are using FreeBSD,
You can always get 256 colors support for vim if you are using Linux/macOS/Cygwin/Android.
With GUI
If you're using terminal emulator, such as gnome-terminal/xterm/iTerm2/mintty/Termux(Android)
Add this code to your .bash_profile or .bashrc or .zshrc
if [[ "$TERM" == "xterm"* ]]; then
  export TERM=xterm-256color
fi

Without GUI
If you're using tty, no GUI, no X server, such as Ubuntu Server/Arch linux/Manjaro.
You can still get 256 colors by installing tmux and fbterm.
Add this code to your .bash_profile or .bashrc or .zshrc
# Check if fbterm is installed
if [ $(command -v fbterm) ] ; then
      exec fbterm -- bash -c 'TERM=fbterm tmux'
else
      exec tmux
fi

FreeBSD without GUI
No solution. you have to remove color themes in .vimrc .

Closing due to no update and great sum up of @j16180339887.


Fix CollisionShape{2D,3D}.debug_color inconsistencies

Fixes #100088.
See also https://github.com/godotengine/godot/pull/39072#issuecomment-1231891794.

CC @BattyBovine

Thanks!


chore(templates/tests): bump spin fileserver component to v0.2.0

Ref https://github.com/fermyon/spin-fileserver/releases/tag/v0.2.0

Looks like a legit error from the assets-test using the fileserver component; will take a look.

The test failure is around the expected response body when a non-existent file is requested.  Currently, we expect "Not Found" as the resp body, however with the latest v0.2.0 version of the fileserver, we get an empty body.  The resp code is still 404 for both:
spin_static_fs.wasm v01.0
$ curl -i http://<IP_ADDRESS>:3000/static/thisshouldbeexcluded/4
HTTP/1.1 404 Not Found
content-length: 9
date: Tue, 14 Nov 2023 17:42:50 GMT

Not Found

spin_static_fs.wasm v0.2.0
$ curl -i http://<IP_ADDRESS>:3000/static/thisshouldbeexcluded/4
HTTP/1.1 404 Not Found
cache-control: max-age=60
etag: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
transfer-encoding: chunked
date: Tue, 14 Nov 2023 17:39:58 GMT

I'm assuming this is due to v0.2.0 using different http libs per wasi-http?  Do we want to add logic to return a "Not Found" body or is this an acceptable change?  cc @dicej

Sorry, I didn't see this earlier.  We can definitely add response bodies in this case -- probably a good idea from a user-friendliness perspective.

Thanks @dicej.  Will close this PR and re-open with a forthcoming 0.2.1 release.


added typescript declaration file

Added typescript declaration file to root
Referenced it in package.json

@Keimeno will be released soon

Released as part of v1.4.0


Windows Terminal Preview Version: 1.17.1023 does not automatically create a profile for PowerShell 7.4.0-preview.3

Windows Terminal version
1.17.1023
Windows build number
10.0.22621.0
Other Software
PowerShell 7.4.0-preview.3 installed from PowerShell-7.4.0-preview.3-win-x64.msi
Steps to reproduce

Install PowerShell 7.4.0-preview.3 from PowerShell-7.4.0-preview.3-win-x64.msi
close Windows Terminal
launch Windows Terminal

Expected Behavior
A new profile for PowerShell 7.4.0-preview.3 should be created
Actual Behavior
No profile is created -- settings.json remains unchanged
settings.zip

If you previously had a PowerShell Preview version installed, and launched terminal + saved your settings without it, you're probably seeing #14061
What happens if you delete state.json from the same folder as settings.json?

Bingo! I deleted both state.json and elevated-state.json (JIC) and the profile got created:

settings.zip

Perfect, thanks! This is definitely a /duplicate of #14061.


Support for new manipulators?

I would like to implement a VideoManipulator class however the PerformConversions job does not support any manipulators other than ImageManipulator.
Maybe it would work something like this:
use Intervention\Image\Image;
use Optix\Media\Facades\Conversion;
use My\VideoManipulator;
use My\Video;

class AppServiceProvider extends ServiceProvider
{
    public function boot()
    {
        // register your manipulators
        // ImageManipulator could be the default
        Conversion::registerManipulators([
            'image' => ImageManipulator::class,
            'video' => VideoManipulator::class
        ]);

        // register by key
        Conversion::register('image', 'thumb', function (Image $image) {
            return $image->fit(64, 64);
        });

        // register by class
        Conversion::register(VideoManipulator::class, 'preview', function (Video $video) {
            return $video
                ->mute() // mute audio
                ->clip(0, 2); // clip from time 0 for 2 seconds
        });

        // or perhaps infer the manipulator by the function parameter type
        // Image infers ImageManipulator
        Conversion::register('thumb', function (Image $image) {
            return $image->fit(64, 64);
        });

        // Video infers VideoManipulator
        Conversion::register('preview', function (Video $video) {
            return $video
                ->mute() // mute audio
                ->clip(0, 2); // clip from time 0 for 2 seconds
        });
    }
}

Thoughts?

Hi @deevus
Really like this idea, not 100% sure on what the API would look like at first thought. Quite like the idea to add the manipulators via the ConversionRegistry and to have a manipulators key in the config file.
Perhaps the manipulator would be resolved depending on the type of media that gets passed to it.
For example:
'manipulators' => [
    ImageManipulator::class => [
        'extensions' => ['png', 'jpg'], // or...
        // 'mime_types' => ['image/*']
    ],
    VideoManipulator::class => [
        'extensions' => ['mp4']
    ]
]

Alternatively it could be slightly more flexible if the the resolved manipulator is specified by the user. For example:
// 1) All manipulator's should implement a FileManipulator interface.
// 2) The media item would be loaded into the manipulator expected by the callback parameter.
// 3) This class would be responsible for preparing the media item for manipulation, i.e
// passing the image path to Intervention\Image and exposing a manipulation API.
// 4) It would also be responsible for returning a savable resource, such as a stream to
// the job that performs the manipulations.

Conversion::register('preview', function (VideoManipulator $manipulator) {
    return $manipulator->mute()->clip(0, 2);
});

I'll try and find some time to prototype this within the coming weeks, should all be a bit clearer then.

This is good.
One thing to note is that the manipulations should be flexible enough to be able to handle external conversion, such as video encoding using AWS services like Elastic Transcoder or Media Convert.
A closure may be enough for some implementations, but there should also be hooks so that Media manipulation can be marked as done, or failed etc asyncronously.
This is obviously more complex than my simple example but it should be doable.


Remove Tatiana from environments part 1

She's gone 🙀 :mournful_seal:
https://trello.com/c/UYKblFuF/332-gds-leaver-tatiana-stantonian-only-start-on-this-card-from-1-nov

Nobody wanted to approve it @binaryberry 🙈


Attempt to fix build failure in readthedocs

This is hacking to support/demo problems detailed in issue #2903 about readthedocs build failures from taskvine

closing this because it was just a demo for #2903


Feature Request: Tableofcontents

For a Rulebook it would be nice to create the Table of Contents with the "\tableofcontents" command.
The Style should be the same as the rest but only in one Column.
Actually the Table is generated in two Columns.

You can comment out \RequirePackage[toc]{multitoc} in lib/dndsection.sty to revert it to full-width table of contents.

How about adding an option to switch between the two behaviours?

Indeed, see https://github.com/evanbergeron/DND-5e-LaTeX-Template/pull/105.


MVCGrid.Net resets predefined filter values

Hi!
I use MVCGrid.Net like

  <input data-mvcgrid-type="filter" data-mvcgrid-option="datetimepicker1" id="TimeStart" name="TimeStart"<EMAIL_ADDRESS>type="text" class="form-control form-control-sm datetimepicker-input" data-target="#datetimepicker1" />

And JavaScript
$(function () {
       $('#datetimepicker1').datetimepicker({
                locale: 'ru',
        });
});

So I assume to see value provided via @ViewBag.TimeStart but it seems like it resets value once it is loaded.  Would you mind to look at it, please. Thanks!

Hi!
I could resolve my needs like
$('#searchButton').click(function () {
            var startDate = $('#datetimepicker1').data('date');
            var endDate = $('#datetimepicker2').data('date');
            var personalName = $('#fieldPersonalFullName').val();
            var deviceName = $('#fieldDeviceFullName').val();

            console.log(startDate);
            console.log(endDate);
            console.log(personalName);
            console.log(deviceName);

            MVCGrid.setAdditionalQueryOptions('AllDiagsGrid',
                {
                    datetimepicker1: startDate,
                    datetimepicker2: endDate,
                    fieldPersonalFullName: personalName,
                    fieldDeviceFullName: deviceName
                });
        });

So no worries at all, folk!
But anyway would be nice to have option reset/dont reset defined filter values.
Thanks!


missing translation in import CSV preview

In the section to import participants import a CSV file there are two buttons in the preview section that are not translated when OpenSlides is in German:

Fixed with #2530


Fixed #27198 -- QueryDict getlist returns now a new list

Fixed issue where QueryDict getlist() returned original list which was mutable. Now getlist returns a new (mutable) list that doesn't affect QueryDict itself.
https://code.djangoproject.com/ticket/27198

Could you test at the MultiValueDict level? tests/utils_tests/test_datastructures.py

merged in 727d7ce6cba21363470aaefb2dc5353017531be3, thanks!


Improve organization of methods like moveRepo->isWin

Probably should move to Entity and maybe even to a separate one (under /Move namespace).

[x] Extract from MoveRepo.
[x] Move to MoveService.
[ ] Split into Move/WinService.
[ ] Make interface methods and namings more meaningful. Like what are You really implementing with the MoveInterface?
[ ] Rename Move and all related parts to SelectedCell.

Rename Move and all related parts to SelectedCell

[x] Repo + Interface.
[x] Service.
[ ] Tests.
[ ] Exceptions.
[ ] Entity.

Nice explanation about interfaces

https://www.killerphp.com/articles/php-interfaces/#:~:text=Interfaces allow you to define,don't get with classes.
https://www.geeksforgeeks.org/when-do-we-need-interfaces-in-php/


[GLUTEN-1632][CH]Daily Update Clickhouse Version (20231230)

Auto commit by gluten daily build, please check the build status and merge it if it's green.

Run Gluten Clickhouse CI

Run Gluten Clickhouse CI


[SDK] Feature: Export contract deployment utilities

CNCT-2441

PR-Codex overview
This PR introduces new utility functions for managing infrastructure contracts and contract initialization in the thirdweb SDK, enhancing the deployment process for published contracts.
Detailed summary

Added getInitializeTransaction for preparing contract initialization transactions.
Added getOrDeployInfraForPublishedContract to retrieve or deploy infrastructure for published contracts.
Enhanced documentation for several functions, detailing parameters and return values.
Introduced getDeployedCloneFactoryContract to retrieve deployed clone factory instances.
Added getAllDefaultConstructorParamsForImplementation to fetch default constructor parameters for contract deployments.
Updated the getDeployedInfraContract function with detailed parameter descriptions.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

Merge activity

Nov 22, 8:23 PM EST: The merge label 'merge-queue' was detected. This PR will be added to the Graphite merge queue once it meets the requirements.

changelog

changelog

changelog


Also check for paths in 'xlink:href' attribute.

This allows for the possibility of image source being in an xlink:href attribute, which can be used for referencing SVG sprites. It also removes any anchor in the path, because it's used to reference a symbol inside the file. See detailed problem description in this issue.

Thanks for doing the work on this. Might need to refactor that method so that the types of attributes can be extend by each user. This will do for now though.
:+1


Apply PSR2 formatting to all handwritten code

Added namespaces to unit tests, which prompted much of the other changes.

Current coverage is 88.63% (diff: 12.50%)

Merging #55 into master will decrease coverage by 0.12%

@@             master        #55   diff @@
==========================================
  Files            20         20
  Lines          1371       1373     +2
  Methods         154        154
  Messages          0          0
  Branches          0          0
==========================================
  Hits           1217       1217
- Misses          154        156     +2
  Partials          0          0

Powered by Codecov. Last update 79d1daa...4a5b78b

LGTM


Link filenames in statements to file content

Add option to link filenames in statements to file content (files in the description/media folder). This way, users can inspect the file content from the feedback table.

Example: submission 6607997
Here's how the file content is displayed in the feedback table of the Python judge:

Example: submission 6493363

Instead of all files in workdir we could also put them in evaluation (e.g. evaluation/workdir). This would not make the files accessible by default (the evaluation dir is only accessible for the judge, not for the submitted code; should be treated as readonly). We could then "register" the files that are needed in a given context, which could server multiple purposes:

copy only those files to the execution context (from evaluation/workdir or workdir); this would have the benefit that one context could not remove, change or tamper the files in a ways that influence other contexts
on the feedback page: link filenames in statements to file content
make files available e.g. to Python Tutor (future work)

To zoom in on "registering" the files, some possibilities I can think of:

Include a list of files in the test plan in a special input (e.g. there is a fixed file or you don't want to work with arguments)
Introduce a new "type" in the function arguments: "FileName".

Works with how we often do things (pass the "path" to the file as an argument)
Allows to annotate those params as a file for Dodona (I forget how this happens, but it will be cleaner than guessing which params are a file and which not)

Both could be supported at the same time.

This is how it looks like in a testcase on the feedback table:

coordinates('photo01.txt')

return

{(0, 0), (1, 1), (0, 2)}

Maybe we could also experiment with providing file links in a separate context-level message in the feedback table.


How do I send multiple Jobs in one Queue or run multiple Queues

Hi there, first of all I'm new to node js and adonis so bear with me please. So I was trying to make some heavy process running on queue, including send emails and write some history logs.

Event.on('approve::spending_house', async (parameters)=>{
console.log('HISTORY LOG & EMAIL EVENT FIRED!')
await Queue.select('high').andDispatch(new SendEmailApprovePerHouse(parameters.arrayDeptIdEmail, parameters.arrayHouseIdEmail, parameters.year, parameters.logModel, parameters.sendEmailPerBO, parameters.divisi_id))
})

as you can see I created a Job named SendEmailApprovePerHouse it was working fine. But when I add another Job and put it in different event, it shows error Cannot call Queue#process twice.
This is how I defined the new Job in another event:

Event.on('approve::spending_house', async (parameters)=>{
console.log('HISTORY LOG & EMAIL EVENT FIRED!')
await Queue.select('high').andDispatch(new SendEmailApprovePerHouse(parameters.arrayDeptIdEmail, parameters.arrayHouseIdEmail, parameters.year, parameters.logModel, parameters.sendEmailPerBO, parameters.divisi_id))
})
Event.on('reject::spending_house', async (parameters)=>{
console.log('HISTORY LOG & EMAIL EVENT FIRED!')
await Queue.select('high').andDispatch(new SendEmailRejectPerHouse(parameters.arrayDeptIdEmail, parameters.arrayHouseIdEmail, parameters.year, parameters.logModel, parameters.sendEmailPerBO, parameters.divisi_id))
})`

How can I add another Job or Queue?
Thanks.

Hey @yusuf1494 , thanks for logging this issue :) . I see you are logging a bug. However, you didn't follow our template format for reporting bugs delineated here. Please, help us by repurposing your issue to conform with this template format.
Provide the version of NodeJS you are using.
Provide the version of this library which you are using
And all other pertinent info, requested by our issue template
Thank you.

Hello @yusuf1494 , Happy New Year!! Please provide more information to help us make the library better... Thanks 😃

Hello @yusuf1494 , Happy New Year!! Please provide more information to help us make the library better... Thanks 😃


Run issue "No module named 'SoapySDR'"

After installing soapy_power using python 3 i get this issue, SoapySDR is installed, along with simplesoapy (via pip), along with all other dependencies except for SciPy, i believe my environment path is set up right.
i installed this on a raspberry pi model 3
Traceback (most recent call last):   File "/usr/local/bin/soapy_power", line 11, in <module>     load_entry_point('soapy-power==1.6.1', 'console_scripts', 'soapy_power')()   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point     return get_distribution(dist).load_entry_point(group, name)   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point     return ep.load()   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load     return self.resolve()   File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve     module = __import__(self.module_name, fromlist=['__name__'], level=0)   File "/usr/local/lib/python3.5/dist-packages/soapy_power-1.6.1-py3.5.egg/soapypower/__main__.py", line 5, in <module>   File "/usr/local/lib/python3.5/dist-packages/SimpleSoapy-1.5.1-py3.5.egg/simplesoapy.py", line 5, in <module> ImportError: No module named 'SoapySDR'
my paths
$echo $PYTHONPATH /usr/local/bin/python3 $ python3 -c "import sys; print(sys.path)" ['', '/usr/local/bin/python3', '/usr/lib/python35.zip', '/usr/lib/python3.5', '/usr/lib/python3.5/plat-arm-linux-gnueabihf', '/usr/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/dist-packages', '/usr/local/lib/python3.5/dist-packages/SimpleSpectral-1.0.0-py3.5.egg', '/usr/local/lib/python3.5/dist-packages/SimpleSoapy-1.5.1-py3.5.egg', '/usr/local/lib/python3.5/dist-packages/soapy_power-1.6.1-py3.5.egg', '/usr/lib/python3/dist-packages']

Issue figured out, i needed to install python3-soapysdr specifically and not just SoapySDR, now it runs well


Release v0.12.0

Suggesting we do another release. Could do an alpha if we wanted, but that doesn't seem necessary.
Here's the changes
Thoughts?

I support a release

Shall we roll out 0.12.0 and 2.15.0 together, @jakirkham?

Will 0.12.0 be released sometime soon? I'm really anticipating #428 being released.

There was a bug encountered with NumPy 1.26+, which would be good to fix: https://github.com/zarr-developers/numcodecs/issues/470
Given Python 3.12 is out, we might want to add that to CI (wouldn't expect other changes needed)
IDK if we want to wait to release for these things or release and then re-release whenever those fixes land

I'd tend to get a release out ASAP (or ASAJIH -- "as soon as John is happy") and then roll another.

On further investigation, looks like the NumPy issue was an issue resolved in main. So a release will fix that problem (more reason to get a release out)
In terms of Python 3.12, things largely worked, but had to fix an entrypoint test issue: https://github.com/zarr-developers/numcodecs/pull/471
Side note: Think it is time for us to drop Python 3.8. If we would rather have a release with Python 3.8 first, then drop Python 3.8 right after, that is fine. That is, as long as CI remains cooperative (there were some issues cropping up recently with Python 3.8 builds in the Python 3.12 PR). Though would also be ok just dropping it in the Python 3.12 PR (as part of this release)

Went ahead and merged PR: https://github.com/zarr-developers/numcodecs/pull/471
Going to kick off the release

Have tagged the release here: https://github.com/zarr-developers/numcodecs/releases/tag/v0.12.0

Seeing some sporadic issues with macOS CI where it tries to use AVX2 instructions. We might want to just turn this off completely. Doubt CI has hardware to cover this instruction. Though unsure why this issue is coming up now
Tracking in issue: https://github.com/zarr-developers/numcodecs/issues/473

Now available on PyPI & conda-forge! 🚀
Closing...

As noted elsewhere, there were a couple bugs that surfaced recently. Most notably issue ( https://github.com/zarr-developers/numcodecs/issues/478 ). Fixes for these are now in
Have cut 0.12.1. Wheels are already up. Conda-forge packages are in progress: https://github.com/conda-forge/numcodecs-feedstock/pull/101

Thanks for your work @jakirkham! Appreciate it! 🙏

Packages are up on conda-forge. Just waiting for them to mirror to CDN. Typically that takes ~15mins after upload (though could be longer)


Add test

@lispyclouds The script in test/script.clj is now a proper runnable babashka test script. It returns a non-zero exit code when the compiled pod doesn't work correctly. Maybe we can hook this up as a test in CI and run it with babashka.

Closed by https://github.com/babashka/pod-babashka-fswatcher/commit/dac58c12a66f3131e70ee914a5c0b3220fc3dc3b


Paths are arbitrarly grouped when SVG is imported

Hi Team!
Thanks a lot for the amazing work, very useful vite plugin.
I noticed that when imported, some of my svg paths were grouped. One of my initial svg is made of one group of 22 paths. Once loaded with the svgLoader(), the svg element becomes a group of 10 paths, some of them being merged arbitrarily. Is it on purpose, or a bug or a feature that I can change?
Not a big deal of course, but keeping distinct path can be useful when animating the svg.
Versions:
"dependencies": {
    "animejs": "^3.2.1",
    "vue": "^3.2.41"
  },
  "devDependencies": {
    "@vitejs/plugin-vue": "^3.2.0",
    "vite": "^3.2.0",
    "vite-svg-loader": "^3.6.0"
  }

Attached are the initial svg (top) and the svg with grouped paths on the page (bottom). Thanks again!

Hi Loris, could you try if the issue is still there if you turn off SVGO? See https://github.com/jpkleemans/vite-svg-loader#disable-svgo

Awesome, that was the reason. I didn't pay attention to this part. Thanks again for the great work!!


Add interface that matches with the InteractsWithQueue trait

Description
This pull request introduces 3 new interfaces - QueueableInterface, DispatchableInterface and InteractsWithQueueInterface. These interfaces provide a clear, consistent way to define the contracts for classes that interact with the queue system in Laravel.
Benefit to the users
One of the main benefits of using interfaces in PHP over traits is that they can be type hinted. This means that when a method or function expects an object of a certain type, you can specify that it must implement a certain interface. This allows for better type safety and makes it easier to catch errors early in the development process. On the other hand, in PHP you cannot type hint a trait, so if you want to make sure a class implements certain methods you need to rely on documentation or conventions. By using interfaces, we can ensure that classes that interact with the queue system in Laravel have a clear and consistent set of methods and behavior and can be type hinted to make sure they are used correctly.
The reasons it does not break any existing features
To avoid any breaking changes, I decided to implement the appropriate interfaces on all classes that already utilize the corresponding traits. This way, these classes will keep their current functionality while also adhering to a clear, consistent contract that aligns with that functionality. In summary, this change will not affect the behavior of existing code and only adds a new way of defining contracts using interfaces.
Paving the way to type safety
I believe this change, the introduction of 3 new interfaces, is a step towards achieving more strongly typed functions in Laravel. In the next major release, developers will have the option to use these interfaces, which may involve breaking changes, to provide explicit types for function parameters, instead of relying on typeless parameters.

If this PR is accepted. I will need advise on how to apply those changes to the 10.x branch and master. Do I cherry-pick my commits into the 10.x branch or a 9.x to 10.x merge occurs once in a while? This part is unclear for me.

@lunfel 9.x is merged into 10.x which is merged into master. No extra action needed.


WR-1430 SSoT database update - updated Skills dataset

Pending a change in the dataset on the ssot side.

data dump has been updated


call filepath.Clean to correct path separator on Windows

TestIsSCMPath fails on Windows because path given may contains slash.

thanks @mattn

No Windows binary in releases page?

Hi @dogancelik  -- I will add later today with @mattn change!
thanks for asking!

@dogancelik @mattn new windows releases are out!
https://github.com/client9/misspell/releases
thanks for your support!
n


[Email Channel] Unable to use email channel

I'm trying to use the email channel, but no matter what I put in the email, I get the following error:
SyntaxError: Unexpected end of input
at Object.parse (native)
at Request._callback (/opt/sumi-dev/node_modules/botbuilder/lib/dialogs/LuisRecognizer.js:70:39)
at Request.self.callback (/opt/sumi-dev/node_modules/botbuilder/node_modules/request/request.js:186:22)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (/opt/sumi-dev/node_modules/botbuilder/node_modules/request/request.js:1081:10)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at IncomingMessage. (/opt/sumi-dev/node_modules/botbuilder/node_modules/request/request.js:1001:12)
at IncomingMessage.g (events.js:260:16)
I'm using botbuilder 3.5.4

Additional detail: this happens from Outlook 365 with the mail format set to HTML. If I set it to plain text, it works.

Further details, it appears the signature is the issue. Attaching the html file.
MCS 2016.zip

Sorry for the delay. I'm trying to get the right people to look at this...

The issue looks to be a bogus response returned from LUIS.

Please attach your BotId if you are still seeing this and we'll take a look in our logs to see if we can isolate the issue.

@jsebrech we need to see the text that we're sending to LUIS. Could you make your bot a little simple like below and then send us the text of the message? Thanks!
var bot = new builder.UniversalBot(connector, function (session) {
     console.log(session.mesage.text);
     session.send('msg: ' + session.message.text);
});

Due to circumstances I won't be able to look into this before the second week of march, but I'll pick it up then.

Ok... we'll leave this open for now

I just now reproduced this issue with the bot with bot handle SumiDevBot and app id edca82d3-9574-4ac7-8b96-6ec3df58b093
The stack trace of the error is the same as in my initial post.
I tested what is being sent as instructed and get this in the logs:
2017-03-09T09:12:42.418Z - info: test
Joeri Sebrechts
MCS
Software & Advisory for
Integrated Real Estate, Facility & Workplace Management
www.MCSsolutions.comhttp://www.mcssolutions.com/
Visit our latest bloghttp://www.mcssolutions.com/be/blog/ | Legal disclaimerhttps://www.mcssolutions.com/be/legal/email-disclaimer/
“We are hiring ! Check herehttps://www.mcssolutions.com/join/?__hstc=36998655.2447b7506e7a91e4d6c5fad7b69c83bd.1462803525775.1464688295311.1464869462405.22&__hssc=36998655.1.1464869462405&__hsfp=2196117068 for open vacancies”

What is your LUIS application ID?

The LUIS app ID is 9fe0a544-65d6-4b1d-a741-273b372aae46

Hitting LUIS directly seems to be working fine.

@jsebrech are you still having the issue with botbuilder v3.7.0?

I am still seeing the issue with v3.7.0. Just reproduced it:
2017-04-27T12:19:07.291Z - error: UniversalBot unexpected error SyntaxError: Unexpected end of input
at Object.parse (native)
at Request._callback (/opt/sumi-dev/node_modules/botbuilder/lib/dialogs/LuisRecognizer.js:70:39)
at Request.self.callback (/opt/sumi-dev/node_modules/botbuilder/node_modules/request/request.js:188:22)
at emitTwo (events.js:87:13)
at Request.emit (events.js:172:7)
at Request. (/opt/sumi-dev/node_modules/botbuilder/node_modules/request/request.js:1171:10)
at emitOne (events.js:77:13)
at Request.emit (events.js:169:7)
at IncomingMessage. (/opt/sumi-dev/node_modules/botbuilder/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:260:16)

@jsebrech can you provide a code example that reproduces this issue?

@jsebrech I'm unable to reproduce your error.  I setup a new bot with Azure Bot Service using Node SDK and the Echo Bot template. Configured the email channel using a valid outlook.com email account. Sent test messages to the bot's email address using 1) plain text, and 2) HTML formatted email, both worked without error. The bot responded to the messages without error in both cases.

@nwhitmont Perhaps you can provide me with an email address to send to? The issue was the specific signature that we add below our emails, not just a HTML-formatted mail.

@jsebrech Can you share the code that is triggering the issue?

reopen if you're still experiencing the issue


How to use preparestatement

I want to use preparestatement in insert。
But when I use by spring-r2dbc(https://docs.spring.io/spring-data/r2dbc/docs/1.2.0-M2/reference/html/#r2dbc.datbaseclient.binding) ，it is a long concatenated string 。
e.g.
REPLACE INTO table (id,name) VALUES (1,'Yeauty')

Set option useServerPrepareStatement as enabled. See README.md.
See also #92, #96 .
I'm closing this issue, if there still have problem, please reopen this issue.


OZW log return blank page

Home Assistant Android version:
1.2.0
Android version:
10
Phone model:
Galaxy S10
Home Assistant version:
0.103
Description of problem:
When trying to check the OZW log from the app in Z wave control panel, it only returns a blank page. Same issue for previous app versions.

Isn't the OZW log opened in a new window on desktop?

Yes opens in new window, Same on mobile browser. Just thinking it should work to check the logs on the mobile app to.

This won't work in the app because we are unable to open new windows and share the authentication context. This should be fixed in the frontend in that it should not open the logs in a new window to begin with.

Going to close this as it's not an issue for the Android app.


Connection refused error when trying to add apikey for local IGLU server on Snowplow mini

I created GCP instance (Snowplow Mini 0.12.5 because I couldn't see link to 0.12.6) based on the tutorial. The control panel working fine. It's available, but a few options inside it are not working.
I tried hard reset of instance via GCP with no change.
When trying to restart services through Control Plane: Error while restarting services, you need to hard reset your server
Trying to add Iglu Apikey: Error:dial tcp <IP_ADDRESS>:5433: connect: connection refused
Elasticsearch&iglu-server&Kibana: 502 Bad Gateway
I have found nothing inside logs so far.

It looks like 0.12.5 is corrupted somehow. I tried with 0.11.0 and it's better. The only problem now is ElasticSearch.
When I try to enter Kibana: Unable to connect to Elasticsearch at http://elasticsearch:9200
What can I do?

Hi @piotrstarzynski , indeed Mini 0.12.5 is broken for GCP. We've addressed this in https://github.com/snowplow/snowplow-mini/issues/283. Could you please try 0.12.6? The setup should be the same as for 0.12.5.

How about the correct storage.googleapis.com link to 0.12.6?

Ah, yes, of course. Sorry. Here they are:
"0.12.6-large"   = "https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-12-6-large-1620758445.tar.gz"
"0.12.6-xlarge"  = "https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-12-6-xlarge-1620748932.tar.gz"
"0.12.6-xxlarge" = "https://storage.googleapis.com/snowplow-mini/snowplow-mini-0-12-6-xxlarge-1620748943.tar.gz"

Works now! Thank You


Run tests on PR

We should setup something like Travis CI to run the tests on PR

Yes, we should use CI tools. Next, we need to complete test cases and try to use Travis CI.
Thank you, John.


Add rust bindings for various features

This PR adds rust bindings for:

mutex
poll
condvars
non-basic timers

As well as the following features in the runtime:

udp_accept (the udp version of tcp accept)
IP loopback support

The rust bindings and runtime features can be separated into multiple PRs if that is desirable.

merged in dev branch.


Fix Pickles async circuits

Fixes up CI errors that weren't noticed when merging #15084 because the unit test script was misconfigured.
There are two fixes:

Introduce a global lock on running any async circuits in the Impls.Step snarky instance. This replaces the two previous locking mechanisms we introduced, which only applied per compile call
Revert certain changes in the order things happen in the step prover, so that the vk lookup now happens after the main circuit logic ran (as it was before #15084)

This fixes two unit tests (pickles.ml / "side-loaded") that rely on storing the vk as part of the circuit
To put the lookup back where it is now, I needed to support async exists for requests. This is added to snarky by the sister PR https://github.com/o1-labs/snarky/pull/841

!ci-build-me

!ci-build-me

!ci-build-me

!ci-build-me

The test still fail, but the problem is now apparent:
File "src/lib/staged_ledger/staged_ledger.ml", line 4860, characters 4-3312: Verification keys set in failed commands should not be usable later threw
(monitor.ml.Error (Failure "Snarky's internal state has been clobbered.")
  ("Raised at Stdlib.failwith in file \"stdlib.ml\", line 29, characters 17-33"

!ci-build-me


Change outage window (current one abandoned)

Context
This week's outage has been abandoned so putting in next one for 29 May
Changes proposed in this pull request
Guidance to review

 TechSource outage 15 May - This is now 29 May


RTL6i

LGTM

Rebased with changes.

Rebased with changes.

LGTM

LGTM

By the way, you're missing:

(RTL6i3) Allows name and or data to be null. If any of the values are null, then key is not sent to Ably i.e. a payload with a null value for data would be sent as follows { "name": "click" }

It can be done in another pull request, just wanted to point it out since the title is "RTL6i".

By the way, you're missing:

(RTL6i3) Allows name and or data to be null. If any of the values are null, then key is not sent to Ably i.e. a payload with a null value for data would be sent as follows { "name": "click" }

@tcard Done

LGTM

LGTM

Review status: 0 of 9 files reviewed at latest revision, all discussions resolved.

Comments from the review on Reviewable.io

Reviewed 2 of 3 files at r1, 7 of 7 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from the review on Reviewable.io


Confusion on debian version?

I have been trying to get this install via npm for awhile but it always fails.  What beaglebone image works "out of box" with octalbonescript?

Hi which debian image do you use? I see they updated debian images for beaglebone. I have set a schedule. I will clear most problems around this library by this Sunday including this one. These are low hanging fruits which I will clear for sure.
In past I promised but could not spare enough time. This time we get all issues solved. Any PRs are welcome.

I last checked with this image. Debian 7.8 (BeagleBone, BeagleBone Black - 4GB SD) 2015-03-01. I see that latest one is Debian 8.2. I am upgrading my own board and will upgrade OBS in a few days.

Specifically?
http://debian.beagleboard.org/images/bone-debian-7.8-lxde-4gb-armhf-2015-03-01-4gb.img.xz
I have tried 8.2 and 7.9 and both fail.  Also  I have tried with the "built
in node" and upgraded node too.  There is no documentation on what debian
version works with OBS.  I think that it should be linked to and stated
specifically so people don't fall down the " this  doesn't work" kind of
issue.
I will try 7.8.
On a side note it looks like for 8.2 the paths are wrong.  I have not had
the time to try to fix it, nor do I fully understand if a  pull request
would work.  Meaning, its almost like OBS needs some sort of release
checking to update paths and other "gotchas"
Thanks!
On Tue, Dec 8, 2015 at 12:24 AM, Aditya Patadia<EMAIL_ADDRESS>wrote:

I last checked with this image. Debian 7.8 (BeagleBone, BeagleBone Black -
4GB SD) 2015-03-01. I see that latest one is Debian 8.2. I am upgrading my
own board and will upgrade OBS in a few days.
—
Reply to this email directly or view it on GitHub
https://github.com/theoctal/octalbonescript/issues/48#issuecomment-162769970
.

Ok, I will update the readme and wiki about supported version. It happened because I assumed that kernels wont change that fast.
You can expect new version resolving this by this Sunday.

Thanks!  That would be great.  I have a project that I am short on time and
really really am not that great on C++!  So I would love to use OBS vs lots
of compiled code!
Thanks again Aditya!
On Tue, Dec 8, 2015 at 12:31 AM, Aditya Patadia<EMAIL_ADDRESS>wrote:

Ok, I will update the readme and wiki about supported version. It happened
because I assumed that kernels wont change that fast.
You can expect new version resolving this by this Sunday.
—
Reply to this email directly or view it on GitHub
https://github.com/theoctal/octalbonescript/issues/48#issuecomment-162770669
.

I just tired the link above that I posted but it fails as well.  FYI.  Thanks!

It’s this link: Debian 7.8 (BeagleBone, BeagleBone Black - 4GB SD) 2015-03-01 http://debian.beagleboard.org/images/bone-debian-7.8-lxde-4gb-armhf-2015-03-01-4gb.img.xz
Regards,
Aditya Patadia

On 08-Dec-2015, at 11:34 AM, ril3y<EMAIL_ADDRESS>wrote:
I just tired the link above that I posted but it fails as well. FYI. Thanks!
—
Reply to this email directly or view it on GitHub https://github.com/theoctal/octalbonescript/issues/48#issuecomment-162779255.

@adityapatadia I tried out the version you listed tonight.  Fresh install, did nothing to it and tried  a sudo npm install octalbonescript but this fails also.
Do you have to do something special?  I upgrade node to v.12 to see if that would help.  But it continues to fail.  Any ideas?

Please post errors you get.

On 09-Dec-2015, at 5:58 AM, ril3y<EMAIL_ADDRESS>wrote:
@adityapatadia https://github.com/adityapatadia I tried out the version you listed tonight. Fresh install, did nothing to it and tried a sudo npm install octalbonescript but this fails also.
Do you have to do something special? I upgrade node to v.12 to see if that would help. But it continues to fail. Any ideas?
—
Reply to this email directly or view it on GitHub https://github.com/theoctal/octalbonescript/issues/48#issuecomment-163066332.

I did this from a fresh install from the image you linked in above.  The only thing I did was change the password on the debian user and then sudo -s to root.
root@beaglebone:/home/debian# npm install -g octalbonescript
|

<EMAIL_ADDRESS>install /usr/local/lib/node_modules/octalbonescript/node_modules/i2c
node-gyp rebuild

make: Entering directory /usr/local/lib/node_modules/octalbonescript/node_modules/i2c/build'   CXX(target) Release/obj.target/i2c/src/i2c.o   SOLINK_MODULE(target) Release/obj.target/i2c.node   SOLINK_MODULE(target) Release/obj.target/i2c.node: Finished   COPY Release/i2c.node make: Leaving directory /usr/local/lib/node_modules/octalbonescript/node_modules/i2c/build'

<EMAIL_ADDRESS>install /usr/local/lib/node_modules/octalbonescript/node_modules/epoll
node-gyp rebuild

make: Entering directory /usr/local/lib/node_modules/octalbonescript/node_modules/epoll/build'   CXX(target) Release/obj.target/epoll/src/epoll.o In file included from ../src/epoll.cc:15:0: ../node_modules/nan/nan.h:328:47: error: ‘REPLACE_INVALID_UTF8’ is not a member of ‘v8::String’ cc1plus: warning: unrecognized command line option "-Wno-unused-local-typedefs" [enabled by default] make: *** [Release/obj.target/epoll/src/epoll.o] Error 1 make: Leaving directory /usr/local/lib/node_modules/octalbonescript/node_modules/epoll/build'
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/share/node-gyp/lib/build.js:267:23)
gyp ERR! stack     at ChildProcess.emit (events.js:98:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:809:12)
gyp ERR! System Linux 3.8.13-bone70
gyp ERR! command "nodejs" "/usr/bin/node-gyp" "rebuild"
gyp ERR! cwd /usr/local/lib/node_modules/octalbonescript/node_modules/epoll
gyp ERR! node -v v0.10.29
gyp ERR! node-gyp -v v0.12.2
gyp ERR! not ok
npm WARN This failure might be due to the use of legacy binary "node"
npm WARN For further explanations, please read
/usr/share/doc/nodejs/README.Debian
npm ERR<EMAIL_ADDRESS>install: node-gyp rebuild
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>install script.
npm ERR! This is most likely a problem with the epoll package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls epoll
npm ERR! There is likely additional logging output above.
npm ERR! System Linux 3.8.13-bone70
npm ERR! command "/usr/bin/nodejs" "/usr/bin/npm" "install" "-g" "octalbonescript"
npm ERR! cwd /home/debian
npm ERR! node -v v0.10.29
npm ERR! npm -v 1.4.21
npm ERR! code ELIFECYCLE
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /home/debian/npm-debug.log
npm ERR! not ok code 0
This is the error code.

I asked the question on epoll module. Let's wait for them to respond. I will post a fix when I find one.

I am getting another issue on my BeagleBone after upgrading it to Debian 8.2. I have posted question on epoll module. Hope to get it too resolved soon.

There is also some issue about how permissions are handled in npm. I am updating readme to recommend installing octalbonescript locally on per project basis. Check this issue for more information: https://github.com/fivdi/epoll/issues/17

@adityapatadia above you make the following comment:

This is a known issue for epoll module. Please refer to https://github.com/fivdi/onoff/wiki/Node.js-v0.10.29-and-native-addons-on-the-BeagleBone-Black

I'd just like to note that this comment as it's totally untrue :)
It's not an epoll issue and has nothing to do with the document linked to.
In a later comment you do however link to https://github.com/fivdi/epoll/issues/17 which describes what the problem really is.

@ril3y mentioned in his error log that he is getting ‘REPLACE_INVALID_UTF8’ error. That issue is related to node version 0.10.29. We can also see from his error log that he is running 0.10.29 version of node.
I also updated my comment to reflect that issue is with nodejs and not only in epoll module. I also added my last comment to educate any user as these issues seems very close from error logs.

Your quite correct. Sorry for adding to the confusion.   Without using -g?
thanks

Yes, I could install it on a per project basis by doing npm install octalbonescript in given project folder.

I find npm v2 much better at handling some of these... especially rebuilding things like newer versions of serialport etc.
sudo npm i -g<EMAIL_ADDRESS>
usually get you there.


Release 8.6.1 (Hotfix)

Summary of the Pull Request
OneFuzz Release 8.6.1 (hotfix)
8.6.1
Added

Service: Added feature flag to toggle Azure DevOps work item processing #3353
Service: Requeue Azure DevOps notifications when the feature flag for work item processing is set to 'disabled' #3358

Codecov Report

Merging #3361 (638ff8c) into main (fc4e698) will increase coverage by 0.62%.
Report is 4 commits behind head on main.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #3361      +/-   ##
==========================================
+ Coverage   31.66%   32.29%   +0.62%
==========================================
  Files         308      275      -33
  Lines       37609    31987    -5622
==========================================
- Hits        11909    10330    -1579
+ Misses      25700    21657    -4043

Files Changed
Coverage Δ

...piService/ApiService/Functions/QueueFileChanges.cs
14.92% <0.00%> (-0.70%)
:arrow_down:

src/ApiService/ApiService/OneFuzzTypes/Enums.cs
32.39% <ø> (ø)

...rvice/TestHooks/NotificationOperationsTestHooks.cs
0.00% <0.00%> (ø)

...ce/ApiService/onefuzzlib/NotificationOperations.cs
23.25% <0.00%> (-1.54%)
:arrow_down:

... and 55 files with indirect coverage changes


Smoke/Smoke Planned symbols not picking up date fields

@Dbarnes1 - Smoke/Smoke Planned symbols not picking up date fields:

These symbols are now fixed, thanks Bob!

Verified in #171


Constraint Violation

Followed the example instructions and received " SSH CA constraint violation ". Is there any troubleshooting advice ?

Here is my output:
user@host:~/yubihsm-ssh-tool$ openssl rsa -pubout -in ca.pem -out ca_pub.pem writing RSA key user@host:~/yubihsm-ssh-tool$ ssh-keygen -i -f ca_pub.pem -m PKCS8 >ca.pub user@host:~/yubihsm-ssh-tool$ yubihsm-shell -a put-asymmetric-key --authkey=0x0002 --password=gpg -d masterkey.gpg-i 10 -l "SSH_CA_Key" -c "sign-ssh-certificate" --in ca.pem Using default connector URL: http://<IP_ADDRESS>:12345 Session keepalive set up to run every 15 seconds Created session 0 Stored Asymmetric key 0x000a user@host:~/yubihsm-ssh-tool$ openssl genrsa -out timestamp.pem Generating RSA private key, 2048 bit long modulus .......................................................+++ ................+++ e is 65537 (0x10001) user@host:~/yubihsm-ssh-tool$ openssl rsa -pubout -in timestamp.pem -out timestamp_pub.pem writing RSA key user@host:~/yubihsm-ssh-tool$ pipenv run yubihsm-ssh-tool templ -T timestamp_pub.pem -k 10 -b 36000 -a 36000 -p root user@host:~/yubihsm-ssh-tool$ yubihsm-shell -a put-template --authkey=0x0002 -pgpg -d masterkey.gpg -i 20 -l "SSH_Template" -A template-ssh --in tem pl.dat Using default connector URL: http://<IP_ADDRESS>:12345 Session keepalive set up to run every 15 seconds Created session 0 Stored Template object 0x0014 user@host:~/yubihsm-ssh-tool$ ssh-keygen -t rsa -N "" -f ./id_rsa Generating public/private rsa key pair. Your identification has been saved in ./id_rsa. Your public key has been saved in ./id_rsa.pub. The key fingerprint is: SHA256:V3PhQFhWh212T/5aJ0j4Br3SHrSnKxTbWn0j16giD/I user@host The key's randomart image is: +---[RSA 2048]----+ |           +=.oo.| |          .. o.o*| |           oo o=o| |          +.+o  o| |        S .X = .o| |         .+ @.=o*| |      . .. * =o=o| |       o..+ + .  | |        Eo.o..   | +----[SHA256]-----+ user@host:~/yubihsm-ssh-tool$ pipenv run yubihsm-ssh-tool req -s ca_pub.pem -t timestamp.pem -I user-identity -n username -V -5h:+5h id_rsa.pub Hash is: b'675d701b2848fc9a1964ee309afc547fe68c5d196cd5e2458268f5f65758cc4e' user@host:~/yubihsm-ssh-tool$ yubihsm-shell -a sign-ssh-certificate --authkey=0x0002 --password=gpg -d masterkey.gpg -i 10 --template-id 20 -A rsa-pkcs1-sha256 --in req.dat --out id_rsa-cert.pub Using default connector URL: http://<IP_ADDRESS>:12345 Session keepalive set up to run every 15 seconds Created session 0 Failed to get certificate signature: SSH CA constraint violation Unable to get ssh certificate user@host:~/yubihsm-ssh-tool$

I just re-tested all the steps and it works on my end. I see you're using a different authentication key from the default one (as you should). How was this created? Could you provide the list of capabilities for that authentication key? You can get the list with
yubihsm-shell -a get-object-info --authkey=2 --password=... -t authentication-key -i 2
Also, please format terminal dumps as code and not as quotes. It makes the text unreadable otherwise.

Apologies for the format. This is what I get from that command:
user@host:~$ yubihsm-shell -a get-object-info --authkey=2 --password=... -t authentication-key -i 2 Using default connector URL: http://<IP_ADDRESS>:12345 Session keepalive set up to run every 15 seconds Created session 0 id: 0x0002, type: authentication-key, algorithm: aes128-yubico-authentication, label: "Master Operations Key", length: 40, domains: 1:2:3:4:5:6:7:8:9:10:11:12:13:14:15:16, sequence: 0, origin: imported, capabilities: change-authentication-key:create-otp-aead:decrypt-oaep:decrypt-otp:decrypt-pkcs:delete-asymmetric-key:delete-authentication-key:delete-hmac-key:delete-opaque:delete-otp-aead-key:delete-template:delete-wrap-key:derive-ecdh:export-wrapped:exportable-under-wrap:generate-asymmetric-key:generate-hmac-key:generate-otp-aead-key:generate-wrap-key:get-log-entries:get-opaque:get-option:get-pseudo-random:get-template:import-wrapped:put-asymmetric-key:put-authentication-key:put-mac-key:put-opaque:put-otp-aead-key:put-template:put-wrap-key:randomize-otp-aead:reset-device:rewrap-from-otp-aead-key:rewrap-to-otp-aead-key:set-option:sign-attestation-certificate:sign-ecdsa:sign-eddsa:sign-hmac:sign-pkcs:sign-pss:sign-ssh-certificate:unwrap-data:verify-hmac:wrap-data, delegated_capabilities: change-authentication-key:create-otp-aead:decrypt-oaep:decrypt-otp:decrypt-pkcs:delete-asymmetric-key:delete-authentication-key:delete-hmac-key:delete-opaque:delete-otp-aead-key:delete-template:delete-wrap-key:derive-ecdh:export-wrapped:exportable-under-wrap:generate-asymmetric-key:generate-hmac-key:generate-otp-aead-key:generate-wrap-key:get-log-entries:get-opaque:get-option:get-pseudo-random:get-template:import-wrapped:put-asymmetric-key:put-authentication-key:put-mac-key:put-opaque:put-otp-aead-key:put-template:put-wrap-key:randomize-otp-aead:reset-device:rewrap-from-otp-aead-key:rewrap-to-otp-aead-key:set-option:sign-attestation-certificate:sign-ecdsa:sign-eddsa:sign-hmac:sign-pkcs:sign-pss:sign-ssh-certificate:unwrap-data:verify-hmac:wrap-data
I had created the authentication key with the following:
yubihsm-shell -a put-authentication-key -i 0x0002 -l "Master Operations Key" --new-password=... --capabilities=change-authentication-key:create-otp-aead:decrypt-oaep:decrypt-otp:decrypt-pkcs:delete-asymmetric-key:delete-authentication-key:delete-hmac-key:delete-opaque:delete-otp-aead-key:delete-template:delete-wrap-key:derive-ecdh:export-wrapped:exportable-under-wrap:generate-asymmetric-key:generate-hmac-key:generate-otp-aead-key:generate-wrap-key:get-log-entries:get-opaque:get-option:get-pseudo-random:get-template:import-wrapped:put-asymmetric-key:put-authentication-key:put-mac-key:put-opaque:put-otp-aead-key:put-template:put-wrap-key:randomize-otp-aead:reset-device:rewrap-from-otp-aead-key:rewrap-to-otp-aead-key:set-option:sign-attestation-certificate:sign-ecdsa:sign-eddsa:sign-hmac:sign-pkcs:sign-pss:sign-ssh-certificate:unwrap-data:verify-hmac:wrap-data --delegated=change-authentication-key:create-otp-aead:decrypt-oaep:decrypt-otp:decrypt-pkcs:delete-asymmetric-key:delete-authentication-key:delete-hmac-key:delete-opaque:delete-otp-aead-key:delete-template:delete-wrap-key:derive-ecdh:export-wrapped:exportable-under-wrap:generate-asymmetric-key:generate-hmac-key:generate-otp-aead-key:generate-wrap-key:get-log-entries:get-opaque:get-option:get-pseudo-random:get-template:import-wrapped:put-asymmetric-key:put-authentication-key:put-mac-key:put-opaque:put-otp-aead-key:put-template:put-wrap-key:randomize-otp-aead:reset-device:rewrap-from-otp-aead-key:rewrap-to-otp-aead-key:set-option:sign-attestation-certificate:sign-ecdsa:sign-eddsa:sign-hmac:sign-pkcs:sign-pss:sign-ssh-certificate:unwrap-data:verify-hmac:wrap-data

Alright, that looks correct to me. One more question, what version of device do you have? You can use get-device-info to get that.
Older versions had a problem in validating the various constraints.

Version number is 2.0.0

Ah, I believe that version had a problem. You may want to get in touch with Support to get that resolved.


Adding Megatron-Style input data pipelines

What does this PR do?
Solves #45
The current version of Nanotron's input pipelines is based on Hugging Face Datasets and relies on clm_preprocess, which tokenises and preprocesses the entire dataset at the beginning of the training (linked to the sequence_length, making it even more difficult to reuse the result across different experiments).
I have developed new data input pipelines based on those included in Megatron. Since I didn't want Nanotron to lose its essence, I removed many functionalities that we don't need (such as those related to BERT models pretraining). What I mainly modified is the torch.utils.data.Dataset, and we continue to work with the same Sampler*, Collator* and DataLoader (* I had to modify them slightly), so it doesn't alter the behavior of other modules like the PipelineEngine at all. It also allows us to continue using the previous pipeline based on Hugging Face Datasets, since I added the script run_train_nanoset.py to launch the training with the new pipeline.
Relevant details:

The new input pipelines work with the same files as Megatron. In this PR, I include references to this project to carry out the data preprocessing, although we could include them here as well. The scripts could be simplified, and necessary dependencies could be added.
I included a new configuration in the .yaml file called NanosetDatasetsArgs, which can replace `PretrainDatasetsArgs. You only need to specify the path to the dataset (generated by Megatron's preprocess_data.py, without the extension as they specify) and the distribution of the dataset samples for each of the splits (train, valid, and test) so that it sums up to 1.
The Nanoset will be the new dataset format. It is a lighter version of GPTDataset and MegatronDataset from Megatron.
To build the datasets, we will use the NanosetBuilder, which, based on a NanosetConfig (Contains NanosetDatasetsArgs + other details), will build a Nanoset for each split. In this first version, we only support one dataset file, but I will include the possibility of using multiple files (BlendedNanoset), hence preserving the NanosetBuilder.
Each Nanoset contains an MMapIndexedDataset. This object is found in indexed_dataset.py and comes from fairseq. Megatron also includes it as such.
I have added a page to the documentation with more details about the preprocessing to create the datasets and the internal functioning similar to what is included in Megatron.

I think maybe we should centralize the input data pipelines and perhaps move the dataloader.py file to another location. I also propose moving several functions from this file with the comments # Question:.
To use the Nanoset datasets, you need to specify the data_path and split fields in config.data.dataset in the .yaml file and use the script run_train_nanoset.py in the same way as run_train.py.
data:
  dataset:
    data_path: /mloscratch/homes/solergib/s-ai/nanotron/datasets/llama2/europarl-gpt-llama2_text_document
    split: 949,50,1
  num_loading_workers: 0
  seed: 1234

I've published the wandb logs of the different tests I have carried out, comparing the HF Datasets and the new Nanoset datasets with 1 and 4 GPUs and resuming training from a checkpoint.
This is a first version, I am open to all suggestions you can think of! I have named it Nanoset, but as I said, you are free to change it!
Toni

Amazing job 🚀 Feel free to request reviews once you feel the PR is ready!

Hello! Everything is ready now. I've added the following:

Nanoset: A new dataset inspired by Megatron.
BlendedNanoset: Similar to the previous one, but this one is for mixing different datasets. It allows specifying the weight of each dataset.
MMapIndexedDataset: Both Nanoset and BlendedNanoset contain MMapIndexedDataset, where the data is actually stored (check, as it queries the data using the get method of MMapIndexedDataset).
NanosetBuilder: Responsible for creating the train, valid, and test datasets whether they are Nanoset or BlendedNanoset type.
New NanosetDatasetsArgs to configure the new datasets. I've also added val_steps to TokensArgs to generate valid and train datasets.
Tools for:

Preprocessing the data and generating Megatron-style datasets (tools/preprocess_data.py). Inspired by Megatron-LM/tools/preprocess_data.py, I have developed a simpler version, getting rid of several features we don't need, such as split sentences or multimodal datasets. I've also enforced the tokenizer to be built with AutoTokenizer.from_pretrained. I've kept the input format of Megatron, which is based on a .json file containing the samples to make it easier to transition to Nanotron.
A small script to transform datasets from the Hugging Face Hub (or local) to the .json format needed by tools/preprocess_data.py (tools/hf_datasets_to_json.py).
A tool to merge already preprocessed datasets into a single file (tools/merge_datasets.py).

In the docs, I've added instructions for using the new datasets. As you can see, it maintains compatibility with the original input data pipelines and with the rest of the project. Basically, the only thing I've modified is the torch.utils.data.Dataset, and we continue to work with practically the same collators and samplers and the same dataloader.
Some small comments and doubts that have arisen:

In src/nanotron/dataloader.py, there is the SkipBatchSampler, EmptyInfiniteDataset, and get_dataloader_worker_init that I use in src/nanotron/data/dataloader_builder.py. Perhaps we could move them to nanotron/data/dataloader_builder.py to centralize this section of the project a bit.
In the log_rank function of src/nanotron/logging.py, I've added a try-except. I did this to allow running preprocessing scripts without initializing a distributed process group. We can leave it like this or initialize the distributed process group within the preprocessing files.
In tools/preprocess_data.py, I've added a small trick to preprocess the data without installing Nanotron. Let me know if you think it's interesting or if we should get rid of it.
To preprocess the data with the Hugging Face tokenizers, we would need to add transformers as a dependency of the project.

In short: NanosetBuilder is responsible for building the train, valid, and test datasets. Depending on whether we specify a blend of multiple datasets or not, it will create either a BlendNanoset or a Nanoset for each split. Each Nanoset contains an MMapIndexedDataset, which is the dataset that contains and reads the bytes with the tokens from the files generated by preprocess_data.py, while the Nanoset itself contains "positions" to read from the MMapIndexedDataset for each sample. The BlendNanoset contains one Nanoset with the samples to extract from the MMapIndexedDataset for each specified path, complying with the specified weights.
Looking forward hearing your feedback!
Toni
cc @NouamaneTazi

Hello!
I have simplified the dataset builder, the __getitem__ method from Nanoset, and fixed a minor bug with the parser that couldn't properly identify the NanosetDatasetsArgs.
Toni

@TJ-Solergibert Hi. Thanks for the fantastic PR. Would be cool if you can add some unit tests for build_nanoset_dataloader, and NanosetBuilder(...).build()!

Hello! I've just added the tests.
I tried to stick to the design of the other tests in the repository, but it was also the first time I was developing one 😅. Finally, I've designed a script that tests everything I've included: It starts by creating a .json file like the ones the preprocess_data.py script expects and processes it to generate the .idx and .bin files containing the tokens. Then, we verify that we can create each type of Nanosets (Nanoset and BlendedNanoset) and create the Dataloader. Finally, we check that the content of the batches in each and every process is appropriate.
In this verification (assert_batch_dataloader), we ensure that the content of each element of the batch (input_ids, input_mask, label_ids & label_mask) is exactly the same across processes within the same tensor parallel group, and also that the class of each element (distinguishing between tensors with ids, tensors with masks, and TensorPointer) is exactly the same across processes within the same data parallel group.
I have locally run the pytest on a cluster with up to 8 GPUs, and they have been satisfactory (It hurt a little to waste the GPUs, but for the development of the test, I did it by applying a small patch to be able to start the dist group with "gloo"). I look forward to your comments!
Toni

Hello!
I think it's a good time for a review. I've further cleaned up the project and made sure to thoroughly document the operation of the Nanosets. Below, I provide all the organized information.
Nanoset
Nanosets are a new type of dataset for Nanotron inspired by those of Megatron. They maintain their performance by dispensing with unnecessary features and (trying to) perserving the essence of Nanotron. In essence, it's just a torch.utils.data.Dataset, so we maintain compatibility with the rest of the project by slightly modifying some aspects related to data loading, such as the collator or the DistributedSampler. Inside, each Nanoset has an MMapIndexedDataset from which we extract the tokens to build the samples. The main task of the Nanoset is to control the logic for constructing the samples from the tokens of the MMapIndexedDataset.
BlendedNanoset
The BlendedNanoset is used to create a mixture of Nanosets by specifying the weights for each of them. Essentially, the BlendedNanoset is only responsible of ensuring that the dataset indices comply with the specified blend, as the samples are extracted from the Nanosets.
NanosetDatasetArgs
To use the Nanosets, I have added a new configuration to the config .yaml file (NanosetDatasetsArgs). The user only needs to enter the dataset(s) they want to use, how they want to divide the dataset into train, valid, and test partitions, and optionally a directory to store Nanoset metadata for reusing the same configuration across different runs.
Tools
Added tools for:

Preprocessing the data and generating Megatron-style datasets (tools/preprocess_data.py). Inspired by Megatron-LM/tools/preprocess_data.py, I have developed a simpler version, getting rid of several features we don't need, such as split sentences or multimodal datasets. I've also enforced the tokenizer to be built with AutoTokenizer.from_pretrained. I've kept the input format of Megatron, which is based on a .json file containing the samples to make it easier to transition to Nanotron.
A small script to transform datasets from the Hugging Face Hub (or local) to the .json format needed by tools/preprocess_data.py (tools/hf_datasets_to_json.py).
A tool to merge already preprocessed datasets into a single file (tools/merge_datasets.py).

Docs
In the docs, I've added instructions for preprocessing the data, using the Nanosets and how do the work under the hood. I strongly recommend taking a look at the examples of how the samples are constructed to understand the operation of the Nanosets.
Short summary
In short: NanosetBuilder is responsible for building the train, valid, and test datasets. Depending on whether we specify a blend of multiple datasets or not, it will create either a BlendedNanoset or a Nanoset for each split. Each Nanoset contains an MMapIndexedDataset, which is the dataset that contains and reads the bytes with the tokens from the files generated by preprocess_data.py, while the Nanoset itself contains "positions" to read from the MMapIndexedDataset for each sample. The BlendedNanoset contains one Nanoset with the samples to extract from the MMapIndexedDataset for each specified path, complying with the specified weights.
Tests
I tried to stick to the design of the other tests in the repository, but it was also the first time I was developing one 😅. Finally, I've designed a script that tests everything I've included: It starts by creating a .json file like the ones the preprocess_data.py script expects and processes it to generate the .idx and .bin files containing the tokens. Then, we verify that we can create each type of Nanosets (Nanoset and BlendedNanoset) and create the Dataloader. Finally, we check that the content of the batches in each and every process is appropriate.
In this verification (assert_batch_dataloader), we ensure that the content of each element of the batch (input_ids, input_mask, label_ids & label_mask) is exactly the same across processes within the same tensor parallel group, and also that the class of each element (distinguishing between tensors with ids, tensors with masks, and TensorPointer) is exactly the same across processes within the same data parallel group. Also, in the case of BlendedNanoset, we verify that it is composed of more than one Nanoset and that each Nanoset has enough samples to satisfy the BlendedNanoset.
Python vs CPP
In the last commit, I replaced the functions we were using from CPP with Python functions. As expected and as seen in this notebook, the CPP functions are incredibly faster than the Python ones, even causing the training start to be blocked for minutes. In the case of the function used to create the indices of the BlendedNanoset (build_blending_indices), in Python, we would spend 44 seconds to create a dataset of 1e7 samples and over 7 minutes for 1e8 samples, while in CPP, we would only wait for 0.1 and 1 second, respectively. Regarding the function used in each Nanoset to build samples of sequence length from the documents (build_sample_idx), we spend 35 seconds if we have 1e7 documents (each entry in the json file is considered a document), while the same function in CPP would take 0.5 seconds. I believe it is incredibly worthwhile to use the ones developed in CPP.
Other comments

In src/nanotron/dataloader.py, there is the SkipBatchSampler, EmptyInfiniteDataset, and get_dataloader_worker_init that I use in src/nanotron/data/dataloader_builder.py. Perhaps we could move them to nanotron/data/dataloader_builder.py to centralize this section of the project a bit.
In the log_rank function of src/nanotron/logging.py, I've added a try-except. I did this to allow running the preprocessing scripts without initializing a distributed process group. We can leave it like this or initialize the distributed process group within the preprocessing files.
In tools/preprocess_data.py, I've added a small trick to preprocess the data without installing Nanotron. Let me know if you think it's interesting or if we should get rid of it.
To preprocess the data with the Hugging Face tokenizers, I added transformers to the requirements of the project.

That would be all, tell me what you think!
Toni

Hello!
I have finally polished the two remaining fronts: MMapIndexedDataset and preprocess_data.py.

I have completely redesigned the preprocessing script, simplifying it to the maximum. Now the user can use both Hugging Face Datasets and JSON files similar to those used in other projects like Megatron. Basically, once we build the Dataset, we shard it among the number of workers. Each worker will be responsible for tokenizing the text present in its shard and storing it in a numpy array. Finally, we concatenate all the arrays produced by each worker. This way, we get rid of the trick to import nanotron without installing it, revert to the original log_rank function (which we had modified because it required initializing the distributed group) but, as we are now working with Hugging Face Datasets, we need to add Datasets as a project requirement. One last note: We store the tokens with dtype = np.uint16, which limits us to using tokenizers with vocabularies < 65536. We could change it to int32, although it is true that we would double the space.

I have also redesigned the MMapIndexedDataset, the heart of the Nanosets. Now it is much simpler, as it only contains the np.memmap to read the tokens. It contains the __getitem__ method, although the important one is the get, which allows us to access an offset position of the array and extract length tokens. We don't need more for CausalLM.

And where did all the sample and document index things from the Nanoset go? Actually, we don't need them, as the only thing that interests us is to extract sequence length + 1 tokens from the mmap dataset. This way, we save a lot of time in the process of building the indices. Now we simply focus on how many tokens our mmap array has, based on the number of tokens and the sequence length we compute the total number of samples that we can generate from the array and divide them into train, valid, and test.
What we do keep to access the Nanoset indices correctly is the shuffle index and the same logic we used before to build them (Concatenating them in the train split to ensure that we have enough samples), since it is worth remembering that the 3 splits access the same MMapIndexedDataset, but at different positions.
I have updated the docs and here I have uploaded the logs with the same configuration that I published a few days ago. The only thing that changes is the number of samples per Nanoset split, as we now do the division at the token level and not at the document level. I have also added a new test to verify that we are able to reset the state of the dataloader when restarting training.
Toni

Hello!
Thank you for your comments, I will review them later. Regarding what you mentioned, you have to start the training with run_train_nanoset.py and not run_train.py. If you prefer, I can merge everything.
Toni

Hello!
I have integrated Nanosets into run_train.py. I have added a yaml config file to run experiments. Previously, it will be necessary to download and preprocess two datasets. I have chosen yelp_review_full and HuggingFaceH4/testing_alpaca_small from the Hugging Face Hub, and we use the GPT2 Tokenizer.
We download and preprocess the datasets as follows:
python3 tools/preprocess_data.py \
       --input yelp_review_full \
       --split train \
       --output-prefix datasets/yelp_review_full \
       --pretrained-model-name-or-path gpt2 \
       --num-workers 16

python3 tools/preprocess_data.py \
       --input HuggingFaceH4/testing_alpaca_small \
       --split train \
       --column completion \
       --output-prefix datasets/testing_alpaca_small \
       --pretrained-model-name-or-path gpt2 \
       --num-workers 16

We launch the job with:
torchrun --nproc-per-node 4 run_train.py --config examples/config_nanoset.yaml

I have tested it with a setup with 4 GPUs.
Toni.

Hi!
Throughout this week, I'll check all your recommendations, thank you!
transformers and datasets are necessary for data processing (mainly transformers for the tokenizer) so I've though of creating a new flavour/extension for the Nanosets and including them to the 3d_parallelism_unit_tests.yaml workflow.
Toni

Don't merge this PR; I'll open a new one with a truly Nano dataset 👀


Trouble sending email over Gmail

I have created a basic email sending application. I have tested it locally using the mailbox preview server. However, when I change the configuration to send the email over Gmail, I get errors.

When I use port 1025, the error is:- {:error, {:retries_exceeded, {:network_failure, 'smtp.gmail.com', {:error, :timeout}}}}.
When I use port 465, I get:- {:error, {:retries_exceeded, {:network_failure, 'smtp.gmail.com', {:error, :closed}}}}.
When I use port 587, I get:- {:error, {:no_more_hosts, {:permanent_failure, 'smtp.gmail.com', :auth_failed}}}.

My config.exs looks like this:-
config :my_app, MyApp.Mailer,     adapter: Swoosh.Adapters.SMTP,     relay: "smtp.gmail.com",     username<EMAIL_ADDRESS>    password: "password",     tls: :always,     auth: :always,     port: 587
I enabled access through less secure apps in my Gmail account, but it's still not working. Could somebody tell me what the problem is?

@sh4t4d33p I'm assuming this is no longer an issue and closing. Please reopen if that's not the case.

FYI: I am using mailman elixir and maybe same issue with swoosh, using mailman I got
{:error, {:no_more_hosts, {:permanent_failure, 'smtp.gmail.com', :auth_failed}}}

but the error only happen on production, tried locally working good.
Then I changed the google account to the business account then working nicely.
So I think the problem is coming from SMTP account and (maybe) related to server IP address.

Hello @stevedomin, the mentioned issue is still going on!

@mbaraa please file a new issue with your detailed error messages. Thanks!

@princemaple it's already open in #785, I'll try the mentioned solution first I guess!