Array dimensions exceeded supported range

System Information (please complete the following information):

Model Builder Version (available in Manage Extensions dialog): <IP_ADDRESS>6802
Visual Studio Version: Community 2022 v 4.8.04084

Describe the bug

On which step of the process did you run into an issue: About 1 hour into model training, Model Builder displays the following exception:
2023-05-21 13:50:16.8911 DEBUG Array dimensions exceeded supported range.  at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.EnsureCapacity(Int32 min)  at System.Collections.Generic.List1.Add(T item)
at Microsoft.ML.Trainers.FastTree.DataConverter.ValuesList.Add(Int32 index, Double value)
at Microsoft.ML.Trainers.FastTree.DataConverter.MemImpl.MakeBoundariesAndCheckLabels(Int64& missingInstances, Int64& totalInstances)
at Microsoft.ML.Trainers.FastTree.DataConverter.MemImpl..ctor(RoleMappedData data, IHost host, Double[][] binUpperBounds, Single maxLabel, Boolean dummy, Boolean noFlocks, PredictionKind kind, Int32[] categoricalFeatureIndices, Boolean categoricalSplit)
at Microsoft.ML.Trainers.FastTree.DataConverter.Create(RoleMappedData data, IHost host, Int32 maxBins, Single maxLabel, Boolean diskTranspose, Boolean noFlocks, Int32 minDocsPerLeaf, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeatureIndices, Boolean categoricalSplit)
at Microsoft.ML.Trainers.FastTree.ExamplesToFastTreeBins.FindBinsAndReturnDataset(RoleMappedData data, PredictionKind kind, IParallelTraining parallelTraining, Int32[] categoricalFeaturIndices, Boolean categoricalSplit)
at Microsoft.ML.Trainers.FastTree.FastTreeTrainerBase3.ConvertData(RoleMappedData trainData)  at Microsoft.ML.Trainers.FastTree.FastTreeBinaryTrainer.TrainModelCore(TrainContext context)  at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input)  at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings)  at Microsoft.ML.AutoML.SweepablePipelineRunner.RunAsync(TrialSettings settings, CancellationToken ct)  at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__24.MoveNext() --- End of stack trace from previous location where exception was thrown ---  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.BinaryClassificationExperiment.<ExecuteAsync>d__14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/BinaryClassificationExperiment.cs:line 148 --- End of stack trace from previous location where exception was thrown ---  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)  at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 169 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
Clear description of the problem: I am attempting to train a model using a large dataset of 468,079,637 rows with 8 columns each using Model Builder. Five of the columns are string, which do not exceed 255 characters. The remaining columns are boolean (0, 1) values. The total dataset size is 26.3 GBs. I have configured the training to be on a boolean column, and the dataset is unbalanced such that a majority of the items are of class 0. One column is an index and is ignored in Data -> Advanced data options. I started training for 7200 seconds, during which time there is relatively low memory usage in my system (maximum 31.7 GBs). After about 50 minutes of training, memory usage spikes to use all remaining memory, and the above exception is displayed.

To Reproduce
Steps to reproduce the behavior:
N/A
Expected behavior
There are no Array dimensions exceptions.
Additional context
This is using the Data classification scenario. In Advanced training options, all trainers are enabled.

I noticed today that the Data Preview only detects 43,176,428 rows:

@tcaivano  thanks for reporting the  issue related with big data size, we will take look into it.

@tcaivano
What's the column type of your col1 && col2, looks like they should be categorical.
And looks like you are hitting OOM error in fast tree trainer. This is because fast tree uses in-memory transposer by default. It supports disk transposer but we didn't expose it in model builder for now.
As a work-around, you can disable fast tree, fast forest and only enable lgbm, which based on our experience is the trainer that works best on huge dataset. In the meantime, we will push a fix to model builder to enable in-disk transposer for fast tree.

@LittleLittleCloud Yes, you're right - my mistake, both col1 and col2 should both be categorical, not strings.
I'll try that out soon - I'm currently running some additional tests that are taking up my cycles but I should have some bandwidth within the new few hours.

I'm getting some issues with LGBM too.
 2023-05-23 15:50:43.0820 DEBUG Array dimensions exceeded supported range.    at System.Collections.Generic.List1.set_Capacity(Int32 value)
at System.Collections.Generic.List1.EnsureCapacity(Int32 min)    at System.Collections.Generic.List1.Add(T item)
at Microsoft.ML.Trainers.LightGbm.LightGbmTrainerBase4.GetMetainfo(IChannel ch, Factory factory, Int32& numRow, Single[]& labels, Single[]& weights, Int32[]& groups)    at Microsoft.ML.Trainers.LightGbm.LightGbmTrainerBase4.LoadTrainingData(IChannel ch, RoleMappedData trainData, CategoricalMetaData& catMetaData)
at Microsoft.ML.Trainers.LightGbm.LightGbmTrainerBase4.TrainModelCore(TrainContext context)    at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input)    at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings)    at Microsoft.ML.AutoML.SweepablePipelineRunner.RunAsync(TrialSettings settings, CancellationToken ct)    at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__24.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.BinaryClassificationExperiment.<ExecuteAsync>d__14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/BinaryClassificationExperiment.cs:line 148 --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 169 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)

Hmmm OK, can you try sdca/lbfgs then, they both support streaming training so might work?

No luck with LBFGS:
2023-05-23 17:51:59.9082 DEBUG Array dimensions exceeded supported range.    at System.Array.Resize[T](T[]& array, Int32 newSize)    at Microsoft.ML.Internal.Utilities.ArrayUtils.EnsureSize[T](T[]& array, Int32 min, Int32 max, Boolean keepOld, Boolean& resized)    at Microsoft.ML.Trainers.LbfgsTrainerBase3.TrainCore(IChannel ch, RoleMappedData data)
at Microsoft.ML.Trainers.LbfgsTrainerBase3.TrainModelCore(TrainContext context)    at Microsoft.ML.Trainers.TrainerEstimatorBase2.TrainTransformer(IDataView trainSet, IDataView validationSet, IPredictor initPredictor)
at Microsoft.ML.Data.EstimatorChain1.Fit(IDataView input)    at Microsoft.ML.AutoML.SweepablePipelineRunner.Run(TrialSettings settings)    at Microsoft.ML.AutoML.SweepablePipelineRunner.RunAsync(TrialSettings settings, CancellationToken ct)    at Microsoft.ML.AutoML.AutoMLExperiment.<RunAsync>d__24.MoveNext() --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.ML.ModelBuilder.AutoMLService.Experiments.BinaryClassificationExperiment.<ExecuteAsync>d__14.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/Experiments/BinaryClassificationExperiment.cs:line 148 --- End of stack trace from previous location where exception was thrown ---    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)    at Microsoft.ML.ModelBuilder.AutoMLEngine.<StartTrainingAsync>d__21.MoveNext() in /_/src/Microsoft.ML.ModelBuilder.AutoMLService/AutoMLEngineService/AutoMLEngine.cs:line 169 (Microsoft.ML.ModelBuilder.Utils.Logger.Debug)
No error with sdca, but I've yet to get anything with a greater than 0 F1 score

Action item

add disk-transpose fast-tree trainer in advanced training option

PR in ML.NET complete. Model Builder PR pending. To be released alongside advanced training (tuner) options.


Minor feature additions/improvements

Add Polymer.mix and utils tests (with mixin tests).
Set dom-repeat initialCount default to 50 (makes incremental rendering default, so that users must "opt-out" to worse performance)
Allow single-argument set to provoke changes (facades to notifyPath)

Closing, out of date.


Float deprecation warnings

Type checking ./Quantities/Power.idr
./Quantities/Power.idr:57:10:Use of deprecated name Float
./Quantities/Power.idr:57:10:Use of deprecated name Float
./Quantities/Power.idr:57:10:Use of deprecated name Float
./Quantities/Power.idr:58:9:Use of deprecated name Float
./Quantities/Power.idr:58:9:Use of deprecated name Float
./Quantities/Power.idr:58:9:Use of deprecated name Float
./Quantities/Power.idr:58:9:Use of deprecated name Float
./Quantities/Power.idr:59:9:Use of deprecated name Float
./Quantities/Power.idr:59:9:Use of deprecated name Float
./Quantities/Power.idr:59:9:Use of deprecated name Float
./Quantities/Power.idr:60:5-7:Use of deprecated name Float
./Quantities/Power.idr:60:5-7:Use of deprecated name Float
./Quantities/Power.idr:60:5-7:Use of deprecated name Float
./Quantities/Power.idr:60:5-7:Use of deprecated name Float
./Quantities/Power.idr:59:38:Use of deprecated name Float
./Quantities/Power.idr:59:38:Use of deprecated name Float
./Quantities/Power.idr:59:38:Use of deprecated name Float
./Quantities/Power.idr:59:38:Use of deprecated name Float
./Quantities/Power.idr:59:9:Use of deprecated name Float
./Quantities/Power.idr:59:9:Use of deprecated name Float
Type checking ./Quantities/FreeAbelianGroup.idr
Type checking ./Quantities/Core.idr
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:52:8:Use of deprecated name Float
./Quantities/Core.idr:78:17:Use of deprecated name Float
./Quantities/Core.idr:78:17:Use of deprecated name Float
./Quantities/Core.idr:79:17:Use of deprecated name Float
./Quantities/Core.idr:79:17:Use of deprecated name Float
./Quantities/Core.idr:133:22:Use of deprecated name Float
./Quantities/Core.idr:133:22:Use of deprecated name Float
./Quantities/Core.idr:134:22:Use of deprecated name Float
./Quantities/Core.idr:139:20:Use of deprecated name Float
./Quantities/Core.idr:139:20:Use of deprecated name Float
./Quantities/Core.idr:140:20:Use of deprecated name Float
./Quantities/Core.idr:140:20:Use of deprecated name Float
./Quantities/Core.idr:140:20:Use of deprecated name Float
./Quantities/Core.idr:140:20:Use of deprecated name Float
./Quantities/Core.idr:285:3:Use of deprecated name Float
./Quantities/Core.idr:321:18:Use of deprecated name Float
./Quantities/Core.idr:322:18:Use of deprecated name Float
Type checking ./Quantities/SIPrefixes.idr
Type checking ./Quantities/SIBaseQuantities.idr
Type checking ./Quantities/SIBaseUnits.idr
Type checking ./Quantities/SIDerivedQuantities.idr
Type checking ./Quantities/SIDerivedUnits.idr
Type checking ./Quantities.idr
Type checking ./Quantities/NonSIUnits.idr
./Quantities/NonSIUnits.idr:17:8:Use of deprecated name Float
./Quantities/NonSIUnits.idr:22:6:Use of deprecated name Float
./Quantities/NonSIUnits.idr:27:5:Use of deprecated name Float
./Quantities/NonSIUnits.idr:32:6:Use of deprecated name Float
./Quantities/NonSIUnits.idr:49:7:Use of deprecated name Float
./Quantities/NonSIUnits.idr:69:18:Use of deprecated name Float
./Quantities/NonSIUnits.idr:92:5:Use of deprecated name Float
./Quantities/NonSIUnits.idr:95:9:Use of deprecated name Float
./Quantities/NonSIUnits.idr:100:6:Use of deprecated name Float
./Quantities/NonSIUnits.idr:105:7:Use of deprecated name Float
./Quantities/NonSIUnits.idr:112:5:Use of deprecated name Float
./Quantities/NonSIUnits.idr:182:18:Use of deprecated name Float
./Quantities/NonSIUnits.idr:187:14:Use of deprecated name Float
./Quantities/NonSIUnits.idr:192:19:Use of deprecated name Float
./Quantities/NonSIUnits.idr:197:13:Use of deprecated name Float
./Quantities/NonSIUnits.idr:202:13:Use of deprecated name Float
./Quantities/NonSIUnits.idr:207:11:Use of deprecated name Float
./Quantities/NonSIUnits.idr:212:10:Use of deprecated name Float
./Quantities/NonSIUnits.idr:217:11:Use of deprecated name Float
./Quantities/NonSIUnits.idr:222:11:Use of deprecated name Float
Type checking ./Quantities/ImperialUnits.idr
./Quantities/ImperialUnits.idr:30:6:Use of deprecated name Float
./Quantities/ImperialUnits.idr:35:6:Use of deprecated name Float
./Quantities/ImperialUnits.idr:40:7:Use of deprecated name Float
./Quantities/ImperialUnits.idr:45:9:Use of deprecated name Float
./Quantities/ImperialUnits.idr:50:6:Use of deprecated name Float
./Quantities/ImperialUnits.idr:55:8:Use of deprecated name Float
./Quantities/ImperialUnits.idr:67:7:Use of deprecated name Float
./Quantities/ImperialUnits.idr:70:14:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:86:6:Use of deprecated name Float
./Quantities/ImperialUnits.idr:94:7:Use of deprecated name Float
./Quantities/ImperialUnits.idr:97:6:Use of deprecated name Float
./Quantities/ImperialUnits.idr:100:6:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:111:6:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:116:6:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:121:7:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:126:8:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:138:14:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:143:13:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:171:7:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:176:9:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:181:15:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:186:5:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:192:11:
Use of deprecated name Float
./Quantities/ImperialUnits.idr:200:22:
Use of deprecated name Float
Type checking ./Quantities/Information.idr
./Quantities/Information.idr:12:6:Use of deprecated name Float
./Quantities/Information.idr:19:13:Use of deprecated name Float
Type checking ./Quantities/Screen.idr
./Quantities/Screen.idr:25:20:Use of deprecated name Float
./Quantities/Screen.idr:32:14:Use of deprecated name Float

With latest git revision of Idris. Float should be Double.

Thanks for reporting this!


Add automation to prevent breaking URLs

(This was discussed at one of the meetings, filing an issue so it isn't forgotten.)
Any kind of reorganization can break existing URLs. Ideally any breakage is intentional rather than accidental. Even more ideally, we add some redirect mechanism.
Brian Skinn's sphobjinv library might be useful for inspecting Sphinx inventories.

Sphinx-Rediraffe, which we now use on the devguide, can also automatically check this at least at the page level.

To make sure I understand -- you're wanting to make sure that incoming links from the general internet, that point to no-longer-existing documents due to reorg, get redirected to their correct new locations?

Aha, I didn't realize what you were trying to do from the comment in the meeting notes.
sphinx-rediraffe absolutely looks like what you want, not sphobjinv.


Ali: Precourse, week 1

my fork of module repository

Learning Objectives

GitHub:

[x] 🥚 Creating Issues: You can create a new issue in a GitHub repository using an issue template. You can correctly use labels, milestones, and assignment to make your issue findable.
[x] 🥚 Searching Issues: You can find issues in a repository by label, author, and milestone.
[x] 🥚 Project Boards: You can use issues on a GitHub project board to track your class assignments and help your classmates when they need it.

HYF Workflows:

[ ] 🥚 Asking for Help: You can use the question and help-wanted discussion templates in your class repository to ask for help, and to help your classmates when they are blocked.
[ ] 🥚 Check-Ins: You can use the check-in template to track your module learning objectives, and your weekly progress, and to post a retrospective at the end of the module.
[ ] 🥚 Deliverables: You can create an issue for this module's deliverable and track your progress on the class project board.

Your Computer:

[ ] 🥚 File System Management: You can create and organize a new folder in your personal computer for the materials you will study at HYF.

Visual Studio Code:

[ ] 🥚 Install and manage extensions
[ ] 🥚 Open a folder for editing
[ ] 🥚 Create new files/folders inside of the folder
[ ] 🥚 Edit files and save changes

Personal study plan
Good Reads

About Learning to Code

[ ] Learn to Program in 21 Days
[ ] The First 20 Hours
[ ] Effective Learning
[ ] How to increase focus and productivity
[ ] The WORST Study Habit for Software Developers
[ ] Why this is so important
[ ] ASCII Pronunciation Rules for Programmers
[ ] JavaScript Glossary
[ ] Write Code Every Day
[ ] Keep a Study Journal
[ ] Weekly Review Day

Markdown (for taking pretty notes)

[ ] Tutorial
[ ] [Reference](https://guides.github.com/features/mastering-markdown/

[ ] Trello - Project Management

[ ] Notion - All-in-one Workspace

[ ] Asana - Project Management Tool

[ ]Challenge: Take your notes with Nano!

[ ] One Article
[ ] Another Article
[ ] The Docs

Typing Games

shortcutfoo.com
speedcoder.net
speedtyper.dev
typing.io
how-to-type.com
thepracticetest.com
mltype
Typer

Terminal Games

bashcrawl - clone & play
Terminus -
online
iTerm - online

Git Games

oh my git
learn git branching

CSS Games
Practice your CSS

the CSS diner
learn flexbox

flexbox froggy
flexbox advneture
grid garden
flexbox defense
flexbox zombies

grid attack
Hex Invaders (hex color codes)
pixact.ly (pixels)

Chill JS
Games

Be the Computer

compute-it
little-dot

js.checkio.org
screeps
untrusted
the elevator
warrior.js

Pretty Things

p5.js

examples
online editor
tutorials
There is a ?p5 lens in study-lenses to study programs in your browser

crunchzilla
Khan Academy

tutorial
sandbox

vizhub & d3.js - data visualization
Aframe.js - web VR

Noises

tone.js

Other Cool Programming Things

Sonic Pi

Week 1
I Need Help With:
setting up WSL2
What went well?
Already had VS code and git connected with SSH
What went less well?
setting up node was hard. Had to download version manager nvm and after a lot of effort, was able to run it on my Windows machine.
Lessons Learned
How to use node and command study to launch javascript debugger
Sunday Prep Work


CASSANDRA-19620: Refactor ColumnCondition

The patch is a follow up from CASSANDRA-19341. It moves some of the ColumnCondition logic into Operator and ColumnExpression in order to simplify the logic needed to addition new operators and ensure that they are supported everywhere by Cassandra.

Not a full review yet.

https://github.com/apache/cassandra/pull/3331/commits/1070c8a3c437260b5d49e4c6cbae60f31d9ab835—Caleb suggested we verify that the merge of the bounds is okay for Accord (in regards to CASSANDRA-18100) and that we do not overcomplicate it for them. You are in the chat with Ariel. I will look into that big commit once we confirm it.
The other two commits - the simplifications look great. Thank you!
Did you start fixing some of the null handling? Shouldn't we leave this for the other ticket?

According to github these were all the changes done since last time I reviewed before rebase:
https://github.com/apache/cassandra/compare/be5994550feb605e22b0d1f9e42f4f90e63d0b38..093edabe4fe28abfc4174c29a364f2d0b071c28e
The changes are small and cosmetic and my questions were addressed in the PR.
CI doesn't show any issues on rebase and the test that were changed and failing are now all passing after CASSANDRA-19637/
I will do a proof read on the whole final patch but at the moment it seems we are ready.

Was the test coverage for the isSatisfiedBy methods verified? https://github.com/apache/cassandra/pull/3331#pullrequestreview-2091386258

LGTM, we can commit it if CI also thinks it LGTI :-)


Parameterize DeGrooteFregly2016Muscle passive fiber force curve

Currently, the DGF passive fiber force curve is fairly stiff. It would be nice to parameterize this curve. We've done so in the past but not in the best manner. We would like to have a meaningful parameter such as strain at 1 norm force. However, the current equation we use makes such a parameterization difficult; I don't know if we can get an analytical expression. We could potentially do a root-solve in finalizeFromProperties(), but that doesn't sound great.

@noahggordon this could be a fun algebra project.


docs(node): Add instrux for restoring from snapshot

What changed? Why?
Add instructions to help people who want to restore from snapshot
Notes to reviewers
N/A
How has it been tested?
Localhost
Does this PR add a new token to the bridge?
Are you adding an entry to assets.ts?

[X] No, this PR does not add a new token to the bridge
[ ] Yes, and I've confirmed this token doesn't use a bridge override
[ ] Yes, and I've confirmed this token is an OptimismMintableERC20

If you are adding a token to the bridge, please include evidence of both confirmations above for your reviewers.

(moved this comment from https://github.com/base-org/node/pull/241)
@wbnns FWIW, the instructions you've put here didn't work for me (and apparently not for @ifsayyes in https://github.com/base-org/node/issues/239 either)
When I put the geth dir from the untar'd dir in my data dir, base/optimism's geth node did nothing for 24 hours... so I tried again by placing the snapshots dir (the root of the untar'd dir) in the geth data dir. Things started working, though it's still syncing (I predict 7 days total on huge fast machine), if it's taking that long, maybe the snapshot didn't take... but the first solution (the one you've got in this PR, definitely didn't work for me)

@jotto Heya! Thanks for the quick feedback! Double-checking and will follow-up. 👍

@jotto Hey, checking back in and just wanted to confirm that I was able to successfully restore from snapshot using these instructions.

Download snapshot and untar
Place the geth folder in the archive in your geth-data folder or other folder if the DATA_DIR was changed (make sure nothing else is in there)
Start node

The snapshot is about two days (2515 minutes) old.

Thank you for this confirmation... I don't know why my attempts (plural) failed, but I'll try again knowing someone else could make it work. I'll report back here.

Do you mind pasting some of the log output from the base-node-geth container (as opposed to base-node-node) - so I can see what log output for a successfully running geth instance reading from a snapshot looks like?

Here is the log output from my base-node-geth after dropping the geth dir from tar archive into data dir:
1 hour (and then no more) of "Generating state snapshot" every few seconds, final log entry for this looks like:
INFO [04-25|19:32:16.693] Generating state snapshot                root=614833..66c538 at=fef184..ed9b45 accounts=61,928,762 slots=372,846,964 storage=29.50GiB   dangling=0 elapsed=1h8m1.244s  eta=17.225s
and otherwise "Looking for peers" (it never finds any peers, I've waited up to 48 hours)
So maybe this is correct? Is this what a fully synced archive node looks like? I don't think so because If I query the RPC node for latest block I get 12234499 (seems to be stuck there, the snapshot I used was base-mainnet-archive-1711261943.tar.gz - March 24th.... which is also the date of 12234499)
const latestBlockNumber = await web3.eth.getBlockNumber();
const block = await web3.eth.getBlock(latestBlockNumber, true);
console.log(`Block Number: ${block.number}`);

and I get "Latest synced block behind by: 46930 minutes" from the base RPC node... (32 days ago... date of the tar snapshot)

base-node-node logs look like:
t=2024-04-25T19:37:11+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAkyzvaA1fnSftWSuwZf2qgPErNy4m9uei11mBYaDrww7Rh
t=2024-04-25T19:37:11+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmG8YrXGL7AVbA5oetjagMr7zk4o1KGTrHii71fKpADjgZ
t=2024-04-25T19:37:11+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmG8YrXGL7AVbA5oetjagMr7zk4o1KGTrHii71fKpADjgZ addr=/ip4/<IP_ADDRESS>/tcp/9222
t=2024-04-25T19:37:11+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmG8YrXGL7AVbA5oetjagMr7zk4o1KGTrHii71fKpADjgZ
t=2024-04-25T19:37:11+0000 lvl=info msg="attempting connection" peer=16Uiu2HAkvTfEA2Lrj2WchFL7egmd5nX6HGe6C8CjSBpkhMzCAq4u
t=2024-04-25T19:37:11+0000 lvl=info msg="Received signed execution payload from p2p" id=0xd75291e7499f3cd54e6c65d55d273f9296170252702d1fbd7079c3e4f5487284:13642242 peer=16Uiu2HAm3QKWSTr26LHAtsAxs4LwSQpvgGWkKFjGK6b2jnFdSFrC
t=2024-04-25T19:37:11+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0xd75291e7499f3cd54e6c65d55d273f9296170252702d1fbd7079c3e4f5487284:13642242
t=2024-04-25T19:37:12+0000 lvl=info msg="connected to peer" peer=16Uiu2HAkvTfEA2Lrj2WchFL7egmd5nX6HGe6C8CjSBpkhMzCAq4u addr=/ip4/<IP_ADDRESS>/tcp/9222
t=2024-04-25T19:37:12+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAkvTfEA2Lrj2WchFL7egmd5nX6HGe6C8CjSBpkhMzCAq4u
t=2024-04-25T19:37:12+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm4zgBaJF2xzkNpqLzQbdGgZUQAuuWrgBpaKQMyxSJdCAh
t=2024-04-25T19:37:13+0000 lvl=info msg="connected to peer" peer=16Uiu2HAm4zgBaJF2xzkNpqLzQbdGgZUQAuuWrgBpaKQMyxSJdCAh addr=/ip4/<IP_ADDRESS>/tcp/9222
t=2024-04-25T19:37:13+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAm4zgBaJF2xzkNpqLzQbdGgZUQAuuWrgBpaKQMyxSJdCAh
t=2024-04-25T19:37:13+0000 lvl=warn msg="Derivation process temporary error" attempts=395 err="engine stage failed: temp: failed to fetch blobs: failed to get blob sidecars for L1BlockRef 0x214ad8ec43a9bc1c1280ce43c6ac55d4bc951b683c20cff2541e2933d0f3b510:19501581: expected 6 sidecars but got 0"
t=2024-04-25T19:37:13+0000 lvl=info msg="attempting connection" peer=16Uiu2HAmCzw9ezt1tHGeCqcFbGKrL5o6b6zmCtQDYoCywWZk4K6i
t=2024-04-25T19:37:13+0000 lvl=info msg="connected to peer" peer=16Uiu2HAmCzw9ezt1tHGeCqcFbGKrL5o6b6zmCtQDYoCywWZk4K6i addr=/ip4/<IP_ADDRESS>/tcp/9222
t=2024-04-25T19:37:13+0000 lvl=info msg="Starting P2P sync client event loop" peer=16Uiu2HAmCzw9ezt1tHGeCqcFbGKrL5o6b6zmCtQDYoCywWZk4K6i
t=2024-04-25T19:37:13+0000 lvl=info msg="attempting connection" peer=16Uiu2HAm8g9VXNqe5thUGoywAETRctjqqdeuaCR6Lp2tNfbYa7kr
t=2024-04-25T19:37:13+0000 lvl=info msg="Received signed execution payload from p2p" id=0x7c6af6b4ab46a7b0f5c799cce46a137b18e78f3ee4030498ae473b89c9f3711a:13642243 peer=16Uiu2HAm3VP6HuTR1VMhssD9geFfDiki2CVJGbKV3fPq2UNUL3F5
t=2024-04-25T19:37:13+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x7c6af6b4ab46a7b0f5c799cce46a137b18e78f3ee4030498ae473b89c9f3711a:13642243

Note, I am using OP_NODE_L1_TRUST_RPC=true (because my L1 is Erigon). Maybe the snapshot is somehow incompatible with that setting?

@jotto Heya! Hm, let's take a closer look at your config together. Can you please drop by the Base Discord and open a #developer-support ticket? Please tag me (wbnns) when you do. 👍


AutoGluon Tabular ensembling fails with NaN-related error

Hello,
Just tried using AutoGluon on a Pandas DataFrame with the following:
task.fit(train_data=X_train, label='y', output_directory=models_dir, time_limits=3600, excluded_model_types=['KNN'])

None of the columns in the training data contains NaNs, however the first round of ensembling fails as follows.
Any help appreciated.
Fitting model: weighted_ensemble_k0_l1 ... Training model for up to 360.0s of the 2650.99s of remaining time.
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 856, in _train_and_save
    model = self._train_single(X_train, y_train, model, X_val, y_val, **model_fit_kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 825, in _train_single
    model.fit(X=X_train, y=y_train, **model_fit_kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/abstract/abstract_model.py", line 233, in fit
    self._fit(**kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/weighted_ensemble_model.py", line 23, in _fit
    super()._fit(X, y, k_fold=k_fold, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, compute_base_preds=compute_base_preds, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/stacker_ensemble_model.py", line 129, in _fit
    super()._fit(X=X, y=y, k_fold=k_fold, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/bagged_ensemble_model.py", line 127, in _fit
    model_base.fit(X_train=X, y_train=y, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/abstract/abstract_model.py", line 233, in fit
    self._fit(**kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/greedy_weighted_ensemble_model.py", line 37, in _fit
    self.model = self.model.fit(X_train, y_train, time_limit=time_limit)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/tuning/ensemble_selection.py", line 48, in fit
    self._fit(predictions=predictions, labels=labels, time_limit=time_limit)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/tuning/ensemble_selection.py", line 98, in _fit
    scores[j] = self.metric._optimum - calculate_score(
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/metrics/__init__.py", line 349, in calculate_score
    cprediction = sanitize_array(prediction)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/metrics/util.py", line 11, in sanitize_array
    maxi = np.nanmax(a[np.isfinite(a)])
  File "<__array_function__ internals>", line 5, in nanmax
  File "/home/eco/anaconda3/lib/python3.8/site-packages/numpy/lib/nanfunctions.py", line 434, in nanmax
    res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
Warning: Exception caused weighted_ensemble_k0_l1 to fail during training... Skipping this model.
Traceback (most recent call last):
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 856, in _train_and_save
    model = self._train_single(X_train, y_train, model, X_val, y_val, **model_fit_kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/trainer/abstract_trainer.py", line 825, in _train_single
    model.fit(X=X_train, y=y_train, **model_fit_kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/abstract/abstract_model.py", line 233, in fit
    self._fit(**kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/weighted_ensemble_model.py", line 23, in _fit
    super()._fit(X, y, k_fold=k_fold, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, compute_base_preds=compute_base_preds, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/stacker_ensemble_model.py", line 129, in _fit
    super()._fit(X=X, y=y, k_fold=k_fold, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/bagged_ensemble_model.py", line 127, in _fit
    model_base.fit(X_train=X, y_train=y, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/abstract/abstract_model.py", line 233, in fit
    self._fit(**kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/models/ensemble/greedy_weighted_ensemble_model.py", line 37, in _fit
    self.model = self.model.fit(X_train, y_train, time_limit=time_limit)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/tuning/ensemble_selection.py", line 48, in fit
    self._fit(predictions=predictions, labels=labels, time_limit=time_limit)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/ml/tuning/ensemble_selection.py", line 98, in _fit
    scores[j] = self.metric._optimum - calculate_score(
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/metrics/__init__.py", line 349, in calculate_score
    cprediction = sanitize_array(prediction)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/utils/tabular/metrics/util.py", line 11, in sanitize_array
    maxi = np.nanmax(a[np.isfinite(a)])
  File "<__array_function__ internals>", line 5, in nanmax
  File "/home/eco/anaconda3/lib/python3.8/site-packages/numpy/lib/nanfunctions.py", line 434, in nanmax
    res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
ValueError: zero-size array to reduction operation fmax which has no identity

Could you try the pre-release version of AutoGluon? (Ensure fresh venv is used): https://auto.gluon.ai/dev/install.html
Unfortunately with the error you provided, it is hard to reproduce, so if pre-release does not work you will want to provide the full training logs instead of just the error.

Thanks. Tried again using the pre-release but no luck. Full log below.
Beginning AutoGluon training ... Time limit = 3600s
AutoGluon will save models to "/home/eco/ag_models/"
AutoGluon Version:  0.0.16b20210206
Train Data Rows:    773540
Train Data Columns: 48
Preprocessing data ...
Using Feature Generators to preprocess the data ...
Fitting AutoMLPipelineFeatureGenerator...
	Available Memory:                    887030.47 MB
	Train Data (Original)  Memory Usage: 286.21 MB (0.0% of available memory)
	Inferring data type of each feature based on column values. Set feature_metadata_in to manually specify special dtypes of the features.
	Stage 1 Generators:
		Fitting AsTypeFeatureGenerator...
	Stage 2 Generators:
		Fitting FillNaFeatureGenerator...
	Stage 3 Generators:
		Fitting IdentityFeatureGenerator...
		Fitting CategoryFeatureGenerator...
			Fitting CategoryMemoryMinimizeFeatureGenerator...
	Stage 4 Generators:
		Fitting DropUniqueFeatureGenerator...
	Useless Original Features (Count: 2): ['feature_0', 'feature_1']
		These features carry no predictive signal and should be manually investigated.
		This is typically a feature which has the same value for all rows.
		These features do not need to be present at inference time.
	Types of features in original data (raw dtype, special dtypes):
		('bool', [])  :  1 | ['feature_2']
		('float', []) : 44 | ['feature_3', 'feature_4', 'feature_5', 'feature_6', 'feature_7', ...]
		('int', [])   :  1 | ['hour']
	Types of features in processed data (raw dtype, special dtypes):
		('category', []) :  1 | ['feature_2']
		('float', [])    : 44 | ['feature_3', 'feature_4', 'feature_5', 'feature_6', 'feature_7', ...]
		('int', [])      :  1 | ['hour']
	3.1s = Fit runtime
	46 features in original data used to generate 46 features in processed data.
	Train Data (Processed) Memory Usage: 279.25 MB (0.0% of available memory)
Data preprocessing and feature engineering runtime = 3.56s ...
AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'
	To change this, specify the eval_metric argument of fit()
Automatically generating train/validation split with holdout_frac=0.01, Train Rows: 765804, Val Rows: 7736
Excluded Model Types: ['KNN']
	Found 'KNN' model in hyperparameters, but 'KNN' is present in `excluded_model_types` and will be removed.
	Found 'KNN' model in hyperparameters, but 'KNN' is present in `excluded_model_types` and will be removed.
Fitting model: RandomForestMSE ... Training model for up to 3596.44s of the 3596.43s of remaining time.
	-0.0937	 = Validation root_mean_squared_error score
	258.67s	 = Training runtime
	0.15s	 = Validation runtime
Fitting model: ExtraTreesMSE ... Training model for up to 3303.96s of the 3303.96s of remaining time.
	-0.0939	 = Validation root_mean_squared_error score
	118.36s	 = Training runtime
	0.13s	 = Validation runtime
Fitting model: LightGBM ... Training model for up to 3143.25s of the 3143.24s of remaining time.
	-0.0933	 = Validation root_mean_squared_error score
	4.08s	 = Training runtime
	0.01s	 = Validation runtime
Fitting model: LightGBMXT ... Training model for up to 3137.93s of the 3137.92s of remaining time.
	-0.0929	 = Validation root_mean_squared_error score
	8.67s	 = Training runtime
	0.01s	 = Validation runtime
Fitting model: CatBoost ... Training model for up to 3129.12s of the 3129.11s of remaining time.
Warning: Sign differs between AG metric and CatBoost metric variants: root_mean_squared_error, flipping signs.
	-0.0932	 = Validation root_mean_squared_error score
	31.57s	 = Training runtime
	0.02s	 = Validation runtime
Fitting model: XGBoost ... Training model for up to 3097.52s of the 3097.52s of remaining time.
	-0.093	 = Validation root_mean_squared_error score
	67.42s	 = Training runtime
	0.1s	 = Validation runtime
Fitting model: NeuralNetMXNet ... Training model for up to 3029.99s of the 3029.99s of remaining time.
	nan	 = Validation root_mean_squared_error score
	322.72s	 = Training runtime
	0.28s	 = Validation runtime
Fitting model: NeuralNetFastAI ... Training model for up to 2706.99s of the 2706.98s of remaining time.
/home/eco/anaconda3/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
█
	Ran out of time, stopping training early.
█
	-0.092	 = Validation root_mean_squared_error score
	2789.87s	 = Training runtime
	26.72s	 = Validation runtime
█
Fitting model: WeightedEnsemble_L1 ... Training model for up to 360.0s of the -149.84s of remaining time.
	Warning: Exception caused WeightedEnsemble_L1 to fail during training... Skipping this model.
Traceback (most recent call last):
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/tabular/trainer/abstract_trainer.py", line 895, in _train_and_save
    model = self._train_single(X_train, y_train, model, X_val, y_val, **model_fit_kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/tabular/trainer/abstract_trainer.py", line 865, in _train_single
    model.fit(X_train=X_train, y_train=y_train, **model_fit_kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/abstract/abstract_model.py", line 310, in fit
    self._fit(**kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/ensemble/weighted_ensemble_model.py", line 26, in _fit
    super()._fit(X_train, y_train, k_fold=k_fold, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, compute_base_preds=compute_base_preds, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/ensemble/stacker_ensemble_model.py", line 137, in _fit
    super()._fit(X_train=X_train, y_train=y_train, k_fold=k_fold, k_fold_start=k_fold_start, k_fold_end=k_fold_end, n_repeats=n_repeats, n_repeat_start=n_repeat_start, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/ensemble/bagged_ensemble_model.py", line 141, in _fit
    model_base.fit(X_train=X_train, y_train=y_train, time_limit=time_limit, **kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/abstract/abstract_model.py", line 310, in fit
    self._fit(**kwargs)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/greedy_ensemble/greedy_weighted_ensemble_model.py", line 41, in _fit
    self.model = self.model.fit(X_train, y_train, time_limit=time_limit)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/greedy_ensemble/ensemble_selection.py", line 50, in fit
    self._fit(predictions=predictions, labels=labels, time_limit=time_limit)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/models/greedy_ensemble/ensemble_selection.py", line 100, in _fit
    scores[j] = self.metric._optimum - calculate_score(
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/metrics/__init__.py", line 487, in calculate_score
    cprediction = sanitize_array(prediction)
  File "/home/eco/anaconda3/lib/python3.8/site-packages/autogluon/core/metrics/util.py", line 11, in sanitize_array
    maxi = np.nanmax(a[np.isfinite(a)])
  File "<__array_function__ internals>", line 5, in nanmax
  File "/home/eco/anaconda3/lib/python3.8/site-packages/numpy/lib/nanfunctions.py", line 434, in nanmax
    res = np.fmax.reduce(a, axis=axis, out=out, **kwargs)
ValueError: zero-size array to reduction operation fmax which has no identity
zero-size array to reduction operation fmax which has no identity

Seems the issue is coming from:
Fitting model: NeuralNetMXNet ... Training model for up to 3029.99s of the 3029.99s of remaining time.
	nan	 = Validation root_mean_squared_error score
	322.72s	 = Training runtime
	0.28s	 = Validation runtime

Somehow, the Neural network is getting a NaN score... I've never seen this before and it isn't obviously reproducible on any dataset that I have. Furthermore, I can't even figure out a way to artificially get a NaN score output from autogluon.core.metrics.root_mean_squared_error which is the metric used in your run.
I would be very interested if you could re-run this with high verbosity and limit to only using the neural network model via
TabularPredictor(label).fit(..., hyperparameters={'NN': {}}, verbosity=4)

This would give the per-epoch val_score of the NN, and would help identify how it got a score of NaN.

I've merged a PR which will not save any model that gets a NaN score, to avoid crashes in future: #937
However, we are still investigating the root cause of the issue. It seems to be triggered when a prediction is NaN by the model, and we have only been able to reproduce it one time across 100s of runs.

This has been fixed in #954.


Kdbai v1.4

Thank you for contributing to LangChain!

[ ] PR title: "package: description"

Where "package" is whichever of langchain, community, core, etc. is being modified. Use "docs: ..."  for template changes, "infra: ..." for CI changes.
Example: "community: add foobar LLM"

[ ] PR message: Delete this entire checklist and replace with
This PR updates kdbai vector store to the latest version of the kdbai_client

[ ] Add tests and docs: If you're adding a new integration, please include

a test for the integration, preferably unit tests that do not rely on network access,
an example notebook showing its use. It lives in docs/docs/integrations directory.

[ ] Lint and test: Run make format, make lint and make test from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/

Additional guidelines:

Make sure optional dependencies are imported within a function.
Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests.
Most PRs should not touch more than one package.
Changes should be backwards compatible.
If you are adding something to community, do not re-import it in langchain.

If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17.

Hi Langchain Team,
This PR includes the latest updates for the KDBAI integration, bringing it up to compatibility with client-server version 1.4, which is scheduled for release on Monday. Since this version updat introduces breaking changes to the kdbai API, these updates are essential to ensure continued functionality with the new version.
It would be great if you could review and merge this at your earliest convenience.
Thanks so much for your help!

closing for breaking changes with the recommendation to launch and register an external package for the new verison!


Logger logs incorrect line

The package logs an incorrect line reference when called.
Expected behaviour:
Calling logger.log("I live!") on line 4 of index.ts would output:
2022-08-05T15:15:27.266Z [debug | index.ts:4:8] I live!
Actual behaviour:
Calling logger.log("I live!") on line 4 of index.ts outputs:
2022-08-05T15:15:27.266Z [debug | index.ts:150:53] I live!

The issue seems to be with the fact that logger.log calls logger.debug (since they both perform the same duty), which moves the actual caller a line down in the stack trace.
Fixing this should be a simple matter of just moving the logic for figuring out which line called the logger to the initial logging functions themselves (and not logger.write).

Pushed a bandaid fix in #19, but I should figure out a better way of getting the line for next minor revision.

I don't think the public log function should call debug as that is a public function and from the current TODO statement, it looks like you are attempting to make an exception to a function call.
If you want both functions to have the same behaviour, then point them to the same private function. That woud make getCallSignature break. A solution would be to generate the error at the top public function level and then pass it down to getCallSignature. Then the stack would always be just one level too deep.


why ScatterPlotLayer can not draw point both with fill and border???

is it possible to realize? how should i do?

A common technique is to use two scatterplot layers, one for the fill and one for the outline. It may require an extra step to make sure you don't get Z fighting between the layers.

It' s really a solution. But if i did so, it will draw twice the amount of data. And every point has its's own style. it will be twice amount of cacultation again .It's performance will be greatly affected

is there any other solution?

can anyone help me about this??

@DLCJianyf Have you actually tried and seen performance suffer? How many points are you drawing? We have internal applications that use this technique, with big data sets with great performance.
If it truly is an issue, you could always subclass the ScatterplotLayer and modify the shader.

surely,i have saw it. very thankful  for your tip

got it.

@DLCJianyf You did it, congratulations! It looks really nice with the color gradient.
Although resolution seems a bit low, it looks a little pixelized? Maybe some smoothstep-ing of transparency in the fragment shader could help? Just a tip in case you want to keep improving your custom shaders...

yes, i think it will very useful for me. I will try it

It seems that the code did not support webgl fWidth function..............

fwidth (note the lowercase w) either requires writing your shader in GLSL 3.00 ES and using WebGL2 (Chrome or Firefox only) or using a WebGL1 extension (OES_standard_derivatives).
I believe that luma.gl automatically tries to install that extension, so maybe it is not available on your browser?

It looks better

@DLCJianyf That looks great, delighted to see that you got it all to work!
Would you be interested in contributing your improvements back to deck.gl? You could open a PR on the original ScatterplotLayer (or add a new EnhancedScatterplotLayer in experimental-layers) and we could take a look...

Sorry to reopen this issue, but can I ask how this is achieved? I'm having an issue where two overlapping layers using the same data only renders one instance. For example, the following code will only render the last ScatterplotLayer in the array:
render() {
    const coords = [
      { position: [-79.92974571588147, 40.459702877261044, 0] },
      { position: [-79.92892772595097, 40.459069877880204, 0] },
      { position: [-79.92859965826088, 40.458421518470075, 0] },
      { position: [-79.92753871886694, 40.45849420710192, 0] }
    ];

    const outline = new ScatterplotLayer({
      id: 'outline',
      data: coords,
      getRadius: 30,
      getColor: [0xff, 0xff, 0xff, 0xff],
    });

    const fill = new ScatterplotLayer({
      id: 'fill',
      data: coords,
      getRadius: 15,
      getColor: [0x00, 0x00, 0x00, 0xff],
    });

    // order matters
    const layers = [outline, fill];

    return (
      <DeckGL
        {...this.props.viewport}
        layers={layers}
      />
    );
  }

As a first step, try adding parameters: {depthTest: false} to the second layer:
   const fill = new ScatterplotLayer({
      id: 'fill',
      data: coords,
      getRadius: 15,
      getColor: [0x00, 0x00, 0x00, 0xff],
      parameters: {
        depthTest: false
      }
    });

Looks like just updating to 6.0.1 solved the issue. But thanks regardless!


Alan Wake

At the beginning of the level "The Signal" in the cafe humans when you pass near them, they disappear. D9VK linux

In native Direct3D9, humans become transparent. Native D39 Windows

Software information
Alan Wake (Steam), Ultra setting
System information

GPU: AMD Radeon R9 290
Driver: 4.5 (Compatibility Profile) Mesa 19.2.0-devel (git-486b33558a)
Wine version: Proton 4.11-1
DXVK version: Master 08.08.19

Apitrace file(s)

https://drive.google.com/open?id=1vAYcQI_nXzEH5xSnXBBvY5OYCbHgFGo2

Log files

d3d9.log:
AlanWake_d3d9.log

d3d11.log:

dxgi.log:

The same happens even when using Wine without d9vk for me.  AMD Fury X, Mesa 19.3.0-devel (git-ce76be9933). All characters (4 of them in total) suddenly disappears.

Closed this as it fixes it in the trace.
I would really like to hear if it works for you, and anyone who has an NV card.

This is what I have. They are transparent but with a grid of points and horizontals lines inside.
With WineD3D, there is no transparency at all, so I can't compare.
Also, why is there no sound during the cutscenes with d9vk? oO


fix server_info forwarding

We need to forward quorum and we also need to correctly populate load factor

Seems like server_info contains both already. Closing


Issuing credits to Derek

Issue 722222 credits to @dworthen.

 Follow up
On Tuesday, 27-Feb-24 22:19:09 UTC by Gov4Git dev
Issued 722222 credits to member @dworthen.


set Dropbox in iframe instead of video

I'm not sure why you choose to use <Video> tag for dropbox, I tested with <Iframe> and it seems to work batter, making landscape and portrait video look good.
@KnaperYaden  check if this works better with Kiwi Browser ( only valid if google drive links working ok till now, if not forget it)

I didn't know this will work honestly. I didn't even try.
Can anyone confirm it solves the Kiwi browser issue?

@KnaperYaden in case you want to test this Ivelt-Media-<IP_ADDRESS>.zip


 Your question is much more
likely to be answered if it is
a good question

Description
Questions can often be answered by our documentation. Unable to find an answer
in our docs? We'll try to help. In the meantime, if you answer your own
question, please respond with the answer here so that others may benefit as
well. Better yet, open a PR to expand our docs ^_^

I made a restart to my computer with an updated operating system, then try to access localhost:3000 not working, however, before the restart, it was working fine, how I can fix this problem
Thanks

Please review the latest code in the master branch. I am confident that this issue has been resolved. If it persists, kindly reopen the issue with updated details.
Try viewer-dev.ohif.org instead of viewer.ohif.org
Our viewer.ohif.org is deployed from release branch while viewer-dev.ohif.org is our master  branch
Read more about branch explanations here https://docs.ohif.org/development/getting-started#developing


add Arch Linux support when auto installing deps

The changes have now been merged to nmail master in 264dadb.


[mirror] download indices separately for each suite.

Fixes #21.

@nojima Please take a look.  Test has passed.


ng-show/ng-hide is not working on page load with ocLazyLoad

I am loading a state with ocLazyLoad. The controller of that state has this code in the start.
$scope.test = {show: false};
and in html view I have this
<button class="btn"  ng-show="test.show">Hey There</button>
Now when my view is loaded the button is not hidden.
if I log {{test.show"}} it shows false. But this is not applying to button.
I tried something like this to execute controller when view is loaded
 $scope.$watch('$viewContentLoaded',
            function() {
                $timeout(function() {
                },0);
            });

but nothing is working.
while ng-show/ng-hide is not working ng-if works perfectly. I am not able to understand why ng-show doesn't work and ng-if works. I need to use ng-show/ng-hide

This is not from the oclazyload plugin. This is like how you structure the application. Please use the pluker to reproduce the issue.
If there is view and proper controller (which is loaded by the oclazyload), then $scope will be available in the view.


New divination Card not counted in

The new divination Card (like The Siren) are not counted in the trading tool thus are not post'd on the forum.

I'll get this added for the next release.
In the meantime you can manually add it in data.xml by adding an entry for it under the DivinationCard section

Thanks for the fast response :)

Brilliant, thanks for commenting with the list! I've updated the xml and it'll be included in the next release.


Generate the cli as a stand alone executable

Fixes #2605
Now if we run ./gradlew shadowJarExecutable we can execute detekt like this: ./detekt-cli/build/run/detek. So, if we publish this file in each new release, we can help the users how to install/update detekt.
Problem: I think that this only work on UNIX. I didn't update the documentation for this reason. I don't know how many people uses windows CMD now a days... ktlint ignores this fact completely as you can check here: https://ktlint.github.io/ should we do it too? Or we can say somthing like:
Unix: ./detekt
Windows: java -jar detekt
What do you think?

Codecov Report

Merging #2607 into master will increase coverage by 1.70%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2607      +/-   ##
============================================
+ Coverage     25.29%   26.99%   +1.70%
+ Complexity      396      394       -2
============================================
  Files           379      379
  Lines          7437     6742     -695
  Branches       1225     1224       -1
============================================
- Hits           1881     1820      -61
+ Misses         5427     4793     -634
  Partials        129      129

Impacted Files
Coverage Δ
Complexity Δ

...itlab/arturbosch/detekt/api/internal/Validation.kt
33.33% <0.00%> (-16.67%)
0.00% <0.00%> (ø%)

.../gitlab/arturbosch/detekt/cli/console/Colorizer.kt
61.53% <0.00%> (-9.90%)
0.00% <0.00%> (ø%)

...h/detekt/api/internal/DisabledAutoCorrectConfig.kt
55.55% <0.00%> (-8.09%)
4.00% <0.00%> (ø%)

...otlin/io/gitlab/arturbosch/detekt/core/TaskPool.kt
33.33% <0.00%> (-6.67%)
3.00% <0.00%> (ø%)

...in/io/gitlab/arturbosch/detekt/api/SingleAssign.kt
60.00% <0.00%> (-6.67%)
3.00% <0.00%> (ø%)

...in/kotlin/io/gitlab/arturbosch/detekt/api/Issue.kt
50.00% <0.00%> (-5.56%)
2.00% <0.00%> (ø%)

...n/io/gitlab/arturbosch/detekt/api/ConsoleReport.kt
75.00% <0.00%> (-5.00%)
4.00% <0.00%> (ø%)

...tlab/arturbosch/detekt/api/internal/EmptyConfig.kt
66.66% <0.00%> (-4.77%)
4.00% <0.00%> (ø%)

...n/kotlin/io/gitlab/arturbosch/detekt/api/Entity.kt
62.50% <0.00%> (-4.17%)
0.00% <0.00%> (ø%)

...in/io/gitlab/arturbosch/detekt/cli/OutputFacade.kt
77.77% <0.00%> (-4.05%)
5.00% <0.00%> (ø%)

... and 260 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dba20e7...d462bc5. Read the comment docs.

So, if we publish this file in each new release, we can help the users how to install/update detekt.

Would it be useful to expose the .jar as an artifact of the Github Action run?

Would it be useful to expose the .jar as an artifact of the Github Action run?

I don't know exctly how the github releases work. Or what "expose the .jar" means exactly. But I think that we should post this executable for each release at github. And expose all the jars of a PR and master/snapshots would be great for the early adaopters.

I don't know exctly how the github releases work. Or what "expose the .jar" means exactly.

Github Actions allows to "attach" one or more artifact for every build.
Example here https://github.com/cortinico/kscript-template/actions/runs/74006424 this run has a binary artifact.
More context here https://github.com/marketplace/actions/upload-artifact

Isn't this the same thing the application plugin builds us? A zip containing all jars and executables for windows and unix?
Our installation instruction could look like:
curl detekt-cli.zip
unzip detekt-cli.zip
alias detekt = <path>/detekt-cli/bin/detekt.[sh|bat]

Oh! i didn't know about that zip. Yeah. This PR do something similar but not the same. That zip contains an "installation": the binaries and the libs. With this we have a stand alone executable.
I wanted to merge this to make it easy to add detekt to homebrew. If we have that file I don't care that much about this. But anyway I think that for a user that just want to download detekt and use it the stand-alone executable is more handy.
My ultimate idea is to have something like this:
How to install detekt:

brew (MacOS):
brew install detekt
detekt
chocolatey (Windows):
I have no idea how you install things in chocolatey but some command here
detekt
Unix:
curl url/version/detekt && chmod +x detekt
./detekt
Other option
curl url/version/detekt.zip && unzip detekt.zip
./bin/detekt

This PR will help with the "Unix" part. But we don't really need that one if you don't like it.

favor to ask - for github releases, the last few have not had consistent naming, which prevents users from reliable downloading. Could you commit to using a standard naming scheme?

@ZacSweers you mean the v in front of the version?
We began using the github release plugin which has this pattern by default.
I will leave it that way.


BP_agent full report adding values to UVAST

values defined as uint in the adm are being treated as uvast by ion messing with report displaying
available_storage
last_reset_time
num_pend_fwd
num_pend_dis
num_in_cust
num_pend_reassembly
num_fragmented_bundles
num_fragments_produced
num_bundles_deleted
failed_custody_bundles
failed_custody_bytes
failed_forward_bundles
failed_forward_bytes
abandoned_bundles
abandoned_bytes
discarded_bundles
discarded_bytes
bundles_by_priority
bytes_by_priority
src_bundles_by_priority
src_bytes_by_priority
num_failed_by_reason
endpoint_active
endpoint_singleton
endpoint_policy
endpoint_report

tracked by https://github.com/NASA-AMMOS/anms/issues/112


4575 upgrade vite 3.2.3

Upgraded Vite to v3.2.3 + updated plugins as required/recommended.
I've run through all the scenarios I can think of in dev mode, and everything looks to be working as expected.
If you have any thoughts on things I should try, please let me know.
I also fixed something in the loading of RepositoryObjectsPage. It's nearly unnoticeable when running in prod mode, but has been annoying me for a while with HMR in dev.
Resolves #4575

@eladlachmi one more thing related to the memory issue node have with vite:

Not crashing with my node v18 - so maybe there is a fix for the 16.x branch too
Check the option to disable the map files

We need to add --max-old-space-size=4096 to the relevant code gen step in the Makefile in order not to communicate it on every workflow and slack.

@nopcoder I think I'll take this even one step further and add it to the npm script for build
IMHO it's closest to the "source" of the issue

Removed sourcemaps from the production build. They aren't served anyway and just make the binary larger, since they are embedded. For the dev build I've set sourcemaps to inline, which improved the dev experience in Chrome devtools.

Approve in hope we will find a packaging tool based on non node.js soon.

@nopcoder It's still a ways off, but this looks promising


Dataaccessapi

The Data Access API Upgrade
Data Access functionality to allow simple data retrieval across TorQ processes, see docs/dataaccess.md for full specification.
(The link to the blog in the docs will be updated on release.)

There doesn't seem to be much logging through the code base for the size of it.  If an issue occurs in the code how do you know how far it has progressed or where it is?

There doesn't seem to be much logging through the code base for the size of it. If an issue occurs in the code how do you know how far it has progressed or where it is?

Logging is done at the getdata level
inputparams:@[.dataaccess.checkinputs;inputparams;.requests.error[requestnumber;]];
The theory is if an error occurs it will occur in the checkinputs stage. This error will then propagate into the .dataaccess.stats with the error- would you say this is sufficient?


Advanced mod guide

Clarifies #526
Fixes #525
Addresses some of the requested documentation in #513

@bitonality we are using mdbook for docs which means that;

the new github markdown features like important, tip, caution etc. are not supported by it (as far as I know, I don't recall seeing it in their docs, but they do move fast so maybe they have it now). Otherwise I would've been using them a lot more before...
when adding new files, you also need to update the page tree in SUMMARY.md. it's pretty self explanatory how that works

For either of the above you can manually merge docs-export and docs folders in the repo and download mdbook.exe to put in there so you can run a local copy to see how your changes look in it.


Filename* parameter not supported in Content-Disposition header

I tried the following request with the latest ModSec code with nginx/1.13.5 and CRS 3.0. It looks like the multipart parser does not support extension notation (RFC 5987) for filename parameter. I am getting the following error:
Error:
ModSecurity: Access denied with code 400 (phase 2). Matched "Operator Eq' with parameter 0' against variable REQBODY_ERROR' (Value: 1' ) [file "modsecurity.conf"] [line "51"] [id "200002"] [rev ""] [msg "Failed to parse request body."] [data "Multipart parsing error: Multipart: Invalid Content-Disposition header (-14): form-data; name=document; filename=03CB16644685216C1E818D1369BB5D4B.png; filename*=utf-8''03CB16644685216C1E818D1369BB5D4B.png""] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "<IP_ADDRESS>"] [uri "/"] [unique_id "153022593947.636509"] [ref "v247,1"]
Request:
POST / HTTP/1.1
User-Agent: Fiddler
Accept: /
Cache-Control: no-cache
Postman-Token: 17ccb852-8244-40da-8cae-9464716ff393
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
------WebKitFormBoundary7MA4YWxkTrZu0gW
Content-Disposition: form-data; name=document; filename=03CB16644685216C1E818D1369BB5D4B.png; filename*=utf-8''03CB16644685216C1E818D1369BB5D4B.png"
Content-Type: text/plain
------WebKitFormBoundary7MA4YWxkTrZu0gW--

Any ideas for a workaround?

Would it be possible for someone to look into this issue?

Indeed currently the multipart parser does not take extension notation into account for the filename parameter.
For some reason your request hasn't triggered the error here, but the following example request from RFC 5987 does trigger the error:
POST /test.html HTTP/1.1
Host: example.org
Content-Type: multipart/form-data;boundary=boundary
Content-Length: 186

--boundary
Content-Disposition: form-data; name="field1"

value1
--boundary
Content-Disposition: form-data; name="field2"; filename*=UTF-8''%e2%82%ac%20rates

value2
--boundary--

Tagging as missing feature for libModSecurity.
The simplest "solution" is disabling rule IDs 200002 and 200003 from modsecurity.conf-recommended.
Depending on your setup, you could maybe do that only for part of your website that uses the unsupported feature (e.g. /file-upload/ or upload.php ) so you would still have these rules working for the rest of the website.

I'm having this problem as well in ModSecurity 2.9.1 , and so would love to know when this get's fixed in the 2.9.x branch as well, as 3.x.

Adding to the 3.0.4 release milestone.

Are you going to backport enhancements like this to the 2.9.x branch?

I don't think so @davewichers
Development efforts towards the 2.9.x branch is currently at maintenance level, meaning that security issues and/or major bugs are addressed with higher priority, while new features or minor bug fixes are meant for v3.
But being an Open Source project, everyone is welcomed to send patches/pull requests ;)

Hmmm. It's my understanding that v3 really isn't ready for production use with Apache yet, like 2.9.x is. Is that correct? If not, are there instructions on how to seamlessly/easily upgrade/port from 2.9.x to 3.0.x? I couldn't find anything like that. If it is still Alpha, then seems like fixes like this that reduce false positives should be backported if at all possible.

@davewichers Yes the Apache-connector is in development stage at this moment.
Documentation for the connector and libModSecurity can be found on the links below:
https://github.com/SpiderLabs/ModSecurity-apache/blob/master/README.md
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/README.md
If you have more specific issues/questions about the Apache-connector, please direct them in the project page: https://github.com/SpiderLabs/ModSecurity-apache
As mentioned this an OpenSource project and we continue to accept fixes/enhancements to the 2.9.x branch, but the main focus of the very small team that actually and actively develops ModSecurity is for the v3 branch.
If there's anyone from the community willing to help with the addition (backporting might be easier as the logic is usually the same) of this feature to v2 I'm sure the ModSecurity team will gladly provide support during the development 🙂
Thanks.

Support for filename* is in v3/master as of https://github.com/SpiderLabs/ModSecurity/commit/136db3e582261382b668cc75592be667307dd052


Filter unit scopes while typing unit name

Currently (v0.2.0-beta) we have to enter a unit name in the Name of unit edit box then click the Find Namespaces(s) button before being presented with a list of possible unit scopes.
It would be much friendlier to start to display the list as the user types in the Name of unit edit control, so that the list gets filtered as the user types.

Should the list be filtered every time the user presses a key in the Name of unit edit box, or should there be a limit on how many times the lookup is performed, say doing nothing unless a certain time had elapsed?
Will the filtering need to be done in a background task?
Should the filtering not start until, say, 3 letters have been typed?


Send exec status independent of report permissions

status: waiting for experiment leads to decide on weather this is OK


Unsupported authorization type

Hi,
Thank you for creating this project.
Can you please let me know how I can authenticate in the Swagger UI?
Here's what I am doing right now:

POST username and email to /api/users/login which returns a TOKEN.
I click on the authorize button in the UI and enter Token  TOKEN.

But, when I try out any of the endpoints, it gives me unsupported authorization type error.
Thank you

For those who found this

In other words add Token as prefix before generated token.


[Error/Typo] Use of overlay_path instead of overlay_files

What is the improvement or update to the Wiki you wish to see?
In the example config file, both the Movies and TV libraries specify overlay_path instead of overlay_files

Does the docs page already exist? Please link to it.
https://metamanager.wiki/en/nightly/config/overview/#configuration-file-example

PR #1815 will resolve this

It looks like this issue is no longer an issue

Yup, looks fixed. Thanks for pointing it out!

Doesn't quite look fixed everywhere, the docker walkthrough: https://metamanager.wiki/en/latest/pmm/install/docker/
shows this example:
## This file is a template remove the .template to use the file

libraries:                       # This is called out once within the config.yml file
  Movies:                        # Each library must match the Plex library name
    collection_files:
      - pmm: basic               # This is a file within the defaults folder in the Repository
      - pmm: imdb                # This is a file within the defaults folder in the Repository
      # see the wiki for how to use local files, folders, URLs, or files from git
    remove_overlays: false       # Set this to true to remove all overlays
    overlay_files:
      - pmm: ribbon              # This is a file within the defaults folder in the Repository
      # see the wiki for how to use local files, folders, URLs, or files from git

but links to this template: https://raw.githubusercontent.com/meisnate12/Plex-Meta-Manager/master/config/config.yml.template
which contains:
libraries:                       # This is called out once within the config.yml file
  Movies:                        # These are names of libraries in your Plex
    collection_files:
      - pmm: basic               # This is a file within PMM's defaults folder
      - pmm: imdb                # This is a file within PMM's defaults folder
      # see the wiki for how to use local files, folders, URLs, or files from git
    overlay_path:
      - remove_overlays: false   # Set this to true to remove all overlays
      - pmm: ribbon              # This is a file within PMM's defaults folder
      # see the wiki for how to use local files, folders, URLs, or files from git

Pay attention to the use of remove_overlays, overlay_files and overlay_path – it is pretty mixed up and confusing for new users I'd say.


[Bug]:   I can not set 45 minutes for long break (it jumps evey 10 minutes notby minute or I can not edit this)

Version

[X] I'm using version 1.12.0

Known issues

[X] I've checked Known issues

Existing issues

[X] I've checked Existing issues

What operating system are you using?
Windows
Operating System Version
Windows 10
Reproduction steps

Open settings and try to set long break to 45 minutes.

...
Reproduces how often: [What percentage of the time does it reproduce?]
Expected Behavior
Possibility to set 45 minutes or with gradation with minutes or allow edit value with keyboard - one of solution is enough.
Actual Behavior
I can set 40 or 50 minutes not my favorite 45.
Relevant log output
No response
Preferences
No response
Additional information
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Hi, I've added explanation to readme: https://github.com/hovancik/stretchly#advanced-preferences

Explanation is very good but if you want to fix just allow user to edit with slider.
I am programing many GUI and I think that it should easier.
User need to specify long break time and also postpone +5, +10, +15, minutes ... or skip (maybe +5 -5 min for postpone or other value from preferences).
In this tool I can not specify long break and also postpone.
Maybe I just fix in and compile or write same in Python from scratch with less functionality. I was using Workrave but it stopped working with current Windows (some Windows API change?).

Are you trying to set duration or interval? Interval is after number of mini breaks so it depends on interval of Mini Breaks

I will try ... - I found that fixing gradation is simple.

It is only nee to change this file https://github.com/hovancik/stretchly/blob/trunk/app/preferences.html.
Whatever I do not know how to compile node.js to application that is only problem.

        <input type="range" min="5" max="65" step="5" list="tickmarksMiniBreakEvery" name="microbreakInterval" data-divisor="60000" id="miniBreakEvery">
        <datalist id="tickmarksMiniBreakEvery">
          <option value="10"></option>
          <option value="20"></option>
          <option value="30"></option>
          <option value="40"></option>
          <option value="50"></option>
          <option value="60"></option>
        </datalist>
        <output data-unit="minutes"></output>

I send pull request ... https://github.com/hovancik/stretchly/pull/1239
I do not describe it because list of question are too long to read for this small change and I do not understand some or they are unrelated to change.

I found that changing mini break have some impact on long break so I can set 45 min.
Only one problem which I found that is counts even I am AFK after break so it need to be reset manual.
So if I take longer break (than longer for example 30 minutes) it start count not after key/mouse.

Long Breaks depend on Mini Breaks as I mentioned here:

Are you trying to set duration or interval? Interval is after number of mini breaks so it depends on interval of Mini Breaks

AFK behavior can be set up as well, see more here: https://github.com/hovancik/stretchly#natural-breaks-inactivity-time-
You can also disable idle time detection in preferences

I'm having this issue too where I would like to set long break intervals independent of short breaks – i.e. my short breaks are every 20 minutes but I would like to take a long break every 45 minutes.


Servodyn unable to load libdiscon.so on linux based HPC resource

I am having trouble running OpenFAST with Servodyn and the ROSCO controller, on a set of inputs that runs to competition without servodyn turned on.
The exact error I am receiving is:

FAST_InitializeAll:SrvD_Init:BladedInterface_Init:The dynamic library libdiscon.so could not be loaded. Check that the file exists in the specified location and that it is compiled for 64-bit applications.

I am currently running on Rocky Linux (v9.2) on a remote computer. I am using an installation of the dev branch from conda-forge (build: h2da4d3d_287). I am using the dev branch due to its support of the seastate module. I have tried different file pathing syntax:

quotes
full file path
relative file path
having the libdiscon.so in the same directory as the .fst
having the .so in separate folder
renaming the .so

I have also tried downloading different .so files from the ROSCO release page (v2.9.0, v2.8.0 and v2.7.0) and also different methods for uploading them to my remote machine.
I have confirmed that file path printed in the error matches what is in my servodyn.dat
Here is some further information about the version of OpenFAST I am running.

I have exhausted the extent of my OpenFAST debugging abilities and would greatly appreciate any insight that the community can provide.

@JackUmaine OpenFAST displays the same error for failure to find and failure to load the shared library. Based on your debugging, the path to the ROSCO library is probably correct in the ServoDyn input file and the issue is with loading the library. Loading ROSCO will fail if there are any missing shared system libraries that it depends on. Different Linux distributions may include different libraries or different versions of libraries. It looks like Rocky Linux is a derivative of CentOS, but I don't know what the shared libraries in the GitHub releases were compiled on. You can check the dependencies of by running ldd libdiscon.so and it will list the required libraries and the paths to them on your system. If something is missing, it help you identify the file you need to install.

@deslaughter Apologies for the delayed response. I met with them last Thursday and as of this morning it appears there is a fix and I have access to the gfortran library and the .so appears to be loading and reading the .in file. Thank you for your assistance!


Change: New Type: 'Addition'

Codecov Report
Patch coverage: 100.00% and project coverage change: -0.12 :warning:

Comparison is base (ad3243b) 56.80% compared to head (215f9ed) 56.69%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #41      +/-   ##
==========================================
- Coverage   56.80%   56.69%   -0.12%
==========================================
  Files          16       16
  Lines         514      515       +1
  Branches      117      117
==========================================
  Hits          292      292
- Misses        105      106       +1
  Partials      117      117

Impacted Files
Coverage Δ

src/Seminal/Change.hs
77.14% <100.00%> (+0.67%)
:arrow_up:

src/Seminal/Enumerator/Expressions.hs
57.51% <100.00%> (-0.66%)
:arrow_down:

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Convert to hierarchical tiles ?

Is it possible to use this library to convert obj to a hierarchical (multi level for lod) tileset ?
if yes, is there a example somewhere ?

same question


Connection refused

In my code: await ffmpeg.run("-i","http://someurl","output.mp4"), I got error in this image. But when running manually in terminal: ffmpeg -i "http://someurl" output.mp4, I run successfully. I use ffmpeg.run() instead of fetchFile() method because ffmpeg.run() method can pass command to download the segment of video, for example last 10 seconds of video, but fetchFile() method downloads full video. How to fix Connection refused error.

Hi @brunoluiz, Actually I want to merge video segment and audio segment into output.mp4 file. I tried to merge with input is local file, for example await ffmpeg.run("-ss", "5", "-i", "video.mp4", "-ss", "5", "-i", "audio.mp3", "-c", "copy", "-map", "0:0", "-map", "1:0", "-to", "10", "output.mp4") (merge from 5th to 15th second of video and audio), and it ran successfully. But when I tried to merge with input is url, for example await ffmpeg.run("-ss", "5", "-i", "http://videourl", "-ss", "5", "-i", "http://audiourl", "-c", "copy", "-map", "0:0", "-map", "1:0", "-to", "10", "output.mp4"), I got Connection refused error. I also tried to replace http with https and I got https protocol not found, recompile FFmpeg with openssl, gnutls or securetransport enabled error. I think merge feature is available in ffmpeg.wasm because I ran successfully with input is local file.

I mean, the remote fetch feature. I guess it only works with files loaded to Emscripten file system. Another thing that doesn't work as well is streaming data in the input as well, if I recall correctly.

Hi @brunoluiz I tried to use fetchFile function with remote url: const data = await fetchFile('https://someurl') and it ran successfully. Is there any efficient way to use fetchFile function? For example, that url contains 1 hour long video, how to fetch and play video at the same time? Because I see current fetchFile function fetches the entire video first and then plays it, so performance will decrease.
I have referenced this link: https://github.com/ffmpegwasm/ffmpeg.wasm/blob/master/docs/api.md#fetch-file, but there is no way to pass start time, end time of video need to be loaded


[Issue-3327] Extension - Show incorrect withdrawal information of the Bifrost liquid staking when all accounts mode

Related issue: #3327

🚀 Deployed on https://pr-3345--sw-web-runner.netlify.app


Potential false positive warnings from hot reload

Steps to Reproduce
I was working through the Write Your First App tutorial. I received warnings that some program elements didn't run when the view was reassembled, but the UI seemed to be updated correctly.
For example, when I changed my code from step 2 to step 3, I received the following message:
Performing hot reload...
Reloaded 1 of 430 libraries in 627ms.

Some program elements were changed during reload but did not run when the view was reassembled;
you may need to restart the app for the changes to have an effect.
  • RandomWordsState.didUpdateWidget (lib/main.dart:28)
  • RandomWordsState._widget= (lib/main.dart:28)

In addition to the UI appeared to be fine, I didn't know how relevant RandomWordsState.didUpdateWidget and RandomWordsState._widget were to me, since these two methods were not actually in the code I wrote.
Then, when I changed my code from step 3 to step 4, I got the following message:
Initializing hot reload...
Reloaded 1 of 430 libraries in 915ms.

Some program elements were changed during reload but did not run when the view was reassembled;
you may need to restart the app for the changes to have an effect.
  • RandomWordsState._suggestions (lib/main.dart:22)
  • RandomWordsState._biggerFont (lib/main.dart:24)

Again, the UI seemed to be functioning correctly, including the font size, which the message says the relevant code didn't run.
Flutter Doctor
[✓] Flutter (Channel master, v0.2.9-pre.9, on Mac OS X 10.13.3 17D102, locale en-US)
    • Flutter version 0.2.9-pre.9 at /Users/taodong/Code/flutter_repos/flutter
    • Framework revision 5e9424419e (35 minutes ago), 2018-04-03 10:35:07 -0700
    • Engine revision 04f3c6557c
    • Dart version 2.0.0-dev.43.0.flutter-52afcba357

[!] Android toolchain - develop for Android devices (Android SDK 27.0.0)
    • Android SDK at /Users/taodong/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-27, build-tools 27.0.0
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)
    ✗ Android license status unknown.

[✓] iOS toolchain - develop for iOS devices (Xcode 9.2)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 9.2, Build version 9C40b
    • ios-deploy 1.9.2
    • CocoaPods version 1.4.0

[✓] Android Studio (version 3.1)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Java version OpenJDK Runtime Environment (build 1.8.0_152-release-1024-b01)

[✓] IntelliJ IDEA Community Edition (version 2018.1)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin installed
    • Dart plugin version 181.4203.498

[✓] VS Code (version 1.20.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Dart Code extension version 2.9.0

[✓] Connected devices (1 available)
    • iPhone SE • AD00445E-9D4A-454D-892F-1419FF77D3A1 • ios • iOS 11.2 (simulator)

! Doctor found issues in 1 category.

For more information about diagnosing and reporting Flutter bugs, please see https://flutter.io/bug-reports/.

This appears to be a Dart 2 issue, the front end seems to have introduced these synthetic functions, fields causing the message to show up. Will discuss with the front end team and figure out how to suppress them

After more investigation I found the following issues

source finger prints are not implemented in kernel (dart 2) as a result changes to functions/fields or classes are not tracked correctly
only new classes, functions or fields added will be tracked for changes
kernel seems to add new synthetic functions 'forwarding-stub' into a class and these need to be skipped over when computing changes.

My conclusion is that we should turn off these hot reload warnings in dart2 until these issues are fixed (raises false positives and misses potential correct situations).

https://github.com/flutter/flutter/pull/16520 turns these unused reload messages off and the IDE will take over the functionality of indicated unused changes in the UI.

As I understand it, these messages are gone and won't return in this form, so I think this issue is not relevant any more.


Bump graalvm dep from 22.3.0 to 22.3.2

Closing in favor of https://github.com/okta/okta-cli/pull/455


added a interactive quiz section

I have added a interactive quiz section and at last it will give a result and give a feedback.
glimpse of a quiz:

thank you for this contribution.

Now check is it correct.

I fixed the conflict arising from a recent pull request.
It works great now, merging it!🚀
Thank you for your contribution! Happy Hacktoberfest 😄

All checks are not passed for this pr will it considered in hacktoberfest ??

All checks are not passed for this pr will it considered in hacktoberfest ??

As your pr is merged in the main branch now, It will be considered for hacktoberfest

someone has changes the whole ui of quiz which looks very bad than before.

is there any impact of my contribution so that i can add it to my resume ?


get current page data

I understand that onPartialSend is a feature for large survey. However, it is possible only send the "current" page result?
I have gone though the methods in survey model. I can't find suitable methods. Am I missing sth?

@gstkenpo Please call survey.sendResult method
Thank you,
Andrew

just find my own way.
//Only retrieve the results which are in current page
var partialData = result.data;
Object.keys(result.data).forEach(function(key) {
var name = $.grep(survey.currentPage.questions, function(e){ return e.name == key || key.indexOf(e.name) >= 0; /to cater key likes A1b-Comment/})
if (name.length === 0){
delete partialData[key];
}
});
console.log(JSON.stringify(partialData));


Smoother management of geofences

Is your feature request related to a problem? Please describe.
The new interface in /geo-fences/new is great. Being able to define a geofence interactively is a great feature. Unfortunately that does not work for editing, besides changing the radius.
Nevertheless it might be also helpful sometimes to input lat/long.
And one cannot add a geofence from inside Grafana.
Describe the solution you'd like

add lat/long to the /geo-fences/new screen and show changes to them in the map
allow editing also of the location when editing a geofence
Replace the Google Maps links in Grafana by links to /geo-fences/new
optional 4. add button in /geo-fences/new to jump to Google Maps.

Describe alternatives you've considered
This is just a feature idea / request.
Additional context
in Grafana e.g. in this screen:

I like the third option in particular.
Editing geo-fences is currently not allowed because it requires a few more things to be implemented but in the long term, of course, this would be nice to have.
I wonder what's the reason for 1. and 4.? Have you already discovered that there is a way to search for addresses / POIs?

1: well, no, I was just used to that..
4: just for convenience
3: oh, please, that would be great!!!!
2: umm, yes, thought so. But please keep that in mind for the future.

I wonder what's the reason for 1. and 4.? Have you already discovered that there is a way to search for addresses / POIs?

Eh, yes, you can get the coordinates from Google Maps...

Resolved by the latest commits on master :-)

Hoooray! Excellent. Perhaps worth mentioning somewhere (README?), that you have to set the base URL in the settings to a publicly accessible URL of teslamate?

Yep, will do 👍 It should initialize the base URL on first start. Did that work in your case?

No, it didn't. I was thinking about it also. But which initial value are you setting?

It uses the URL reported by window.location.origin. The call is triggered when visiting the home page / status screen for the first time after restarting.

Any chance to test it again, after I have set the base URL?

Only by settings the base_url in the settings table to NULL. Afterwards visit the home page again. I'd be very interested to hear the results!

Ok, that did the job. Great work!
(might be, that I jumped earlier directly to the settings page...)

Thanks! Good to know. Maybe I'll tweak that part a little.


chore(kapitan/examples): Add easy nginx example

Fixes issue #151

@adrianchifor Now, there is only one jsonnet file. Since, it has only 30 lines of code, so no need for split. Thanks to kube.libjsonnet suggested by you. Kjsonnet is awesome.  :+1:

Could this instead be added to examples/kubernetes?
A lot of things are overlapping with that example, so it would be nicer to instead enhance the kubernetes example we already have with a new component (nginx) and 2 new targets (prod-nginx, dev-nginx) for example (?). That way it also get's tested with the existing tests that build examples/kubernetes as well. This ensures our example is always up to date and actually functional.
Remember that when you update the examples/kubernetes you need to update the test files for it as well.

@uberspot where are prod-nginx, dev-nginx files. I am not able to find them.

@uberspot where are prod-nginx, dev-nginx files. I am not able to find them.

Those are the files you'd add with this PR.

@ubrspot there is already a component named nginx, it is in kadet. So, I can change existing one to nginx-kadet and add new one as nginx-jsonnet.

@adrianchifor should I squash all commits or make a new commit? As examples/nginx folder will be deleted.

Squashing is cleaner. :+1:

@uberspot done the changes

@uberspot review this, plz.

@uberspot regarding #303 users are looking for simple examples, can you review?

@adrianchifor you can also review this PR. I have done all the asked changes.

@uberspot Can u look at this PR? I think it is ready to be merged.


Flag unaccompanied minor

Not for Campout necessarily, but will be important for future camps!
If a person registers a minor without and adult, this should create some kind of an alert. For the registrant it should show a message like:
Minors (anyone under 18) must attend Lark Camp(out) with a parent or guardian who is at least 21 years old. Special paperwork is required for minors who attend with a guardian rather than a parent or legal guardian (contact<EMAIL_ADDRESS>for more information). It must be filled out by the parent, guardian, and minor and returned to Lark Camp by (Sept 30), or the minor’s registration will be canceled with no refund.
It should also trigger an alert or flag for the registrar so that they know they need to follow up.

Also Add guardian field
searchable to match unaccompanied minor with their guardian.
Not necessary for campout, but will be needed for Lark Camp.


Listing the versions of a specific dataset.

Hi,
I had gone through the features of Dat, it looks nice and interesting. I faced an issue when i was trying to list the versions of datasets. Imported two different files to two different datasets in a dat repository, when I list the log using dat log command, all versions in the dat repository are listed.
Is there any option for taking the versions of a specific dataset?

Hey @nessyhamsa!
Every time dat takes a snapshot of the data, it saves all of it at once. You can think of a dataset as a file -- and we don't have the capability to go back in time on just files right now. This is a great feature idea though, and I think other people want it too. Thank you!

Can u please provide a way to get the dataset name used for each version during file import along with log details.
It seems currently the log object contains below details.
{ root: false,
change: 42,
date: Wed Dec 09 2015 14:10:41 GMT+0530 (IST),
version: 'b2df5855c3627753a0776a4c7a7751115fca2e9dbed96511f96d29a9d44e0631',
message: '',
links: [ '7c4af74e78a797ae7f18c9cb589176bd874d09eeed90bfb8906719ce6494e48c' ],
puts: 3,
deletes: 0,
files: 0 }
We just need a dataset along with these details.
We didn't identified from which JS in dat folder updating these details at the time of file import.


Fix Axiom mixin incompatibility

Closes #120

i'm busy as hell right now, i'll give this look once i get a moment (hopefully this weekend)

i did not realise that this PR was a single file, i thought it was much bigger and would take me longer to look at LMAO
anyway LGTM!


Volkswagen not mentioned in supported cars

Hi I was checking out your fork deciding whether it could be a good fit on my 2019 VW Golf.  I saw that it's not in the list of manufacturers at the top of the README, but there is mention of some features like custom stock longitudinal control working for VW MQB.  So I'm a little bit confused, is there support for my golf, or is it not recommended?  Cheers!

Please review the stock OpenPilot wiki: https://github.com/commaai/openpilot/wiki/Volkswagen

Just load up the fork and give it a go, worst that can happen is it doesn't work

I did, it works well.

I see this is marked as completed, but the readme still says:
sunnypilot is recommended to be used for most of the following car makes' models:
Hyundai/Kia/Genesis (HKG)
Honda/Acura
Toyota/Lexus
Subaru
Chrysler/Jeep/RAM (FCA)
General Motors (GM)
Is there a reason not to include VW in that list?

The README is being refactored at the moment, but please feel free to make a PR for the time being to add Volkswagen on there :)


Videos not scanning and nothing in app.log

Device

Switch

Native screen recording and 720 capture

Game language: English
App version: 1.7

** Problem(s) **
I've attempted on 2 PCs and a a Mac, same issue on all three. Given that, I think there is a strong possibility the problem is me. It launches and runs with no errors, the app.log is empty and it doesn't seem to have seen the video. No frames are created. Here's the output given in the window

Input video location:  /Users/Admin/Downloads/utsushis-charm-1.7.0/inputs
Frame location:  /Users/Admin/Downloads/utsushis-charm-1.7.0/frames
Autosaving Charms:  True
Skipping frame extraction:  False
Deleting existing frames:  False
Extracting Frames
Total input files to scan: 0
Reduced frames from 0 to 0
Extracting charms from frames
Language pack was found
Your charms were saved to "charms.json" and "charms.encoded.txt".
To use on MHR-Wiki, you'll want the contents of "charms.encoded.txt"
Charm extration complete

I've attached the recording I pulled form the switch sd card, I've also tried using a 720 capture with the same issue though.
https://user-images.githubusercontent.com/75280988/179432701-416d51af-fcf9-45ed-a0e5-d5e720bbbf25.MP4

When you ran it on a mac, was it from built from source? Have you tried running from source on your PCs? From what I see, your recording seems perfectly fine, I will try to run your clip on my computer to see if it works.
If it doesn't I will try to figure it out during the week.

I was able to extract your charms from your clip. So I definitely think it's a "you" issue.
I saw that you were working in the admin user directory, the app might not be able to access the contents of your "inputs" directory.
In any case, here's what I extracted from your clip:
charms.encoded.txt
I wont do this every time, but if you want to upload all your clips to a google drive or something so I can extract them for you while you figure it out, feel free to go ahead and do so.

Closing because inactive and it seems user-based


Added support for Windows CI/CD

Describe your changes
Added Support for Windows in CI/CD
Issue ticket number and link
Closes #21
Checklist before requesting a review

[X] I have performed a self-review of my code
[X] I have added thorough tests (when necessary).
[X] I have added the right documentation in the docstring and changelog (when needed)

:books: Documentation preview :books:: https://ploomber-engine--73.org.readthedocs.build/en/73/

Windows CI doesn't support Python 3.8 as SQLite don't have support for the json_extract function for Python 3.8 in Windows


[Video Tutorial] Pegswap

Build video on how to make ERC-20 token ERC-677 compatible.
https://pegswap.chain.link/

there is a video . i'll check if it needs to be updated

Hello @fedorbcl
there is actually a video which was made 1 year ago:  https://www.youtube.com/watch?v=WKvIGkBWRUA.
you can also find it here