Issue with overlapping cidr ranges

Yoooo harry im trying to deploy your template and keep getting this error message any ideas?
│ Error: creating EC2 Subnet: InvalidSubnet.Conflict: The CIDR '<IP_ADDRESS>/16' conflicts with another subnet
│       status code: 400, request id: 036a8588-74b0-4bb7-a487-47fe02efcdc7
│
│   with aws_subnet.public_subnet,
│   on main.tf line 12, in resource "aws_subnet" "public_subnet":
│   12: resource "aws_subnet" "public_subnet" {
│

Hi Wesbest,
The module deploys in the default VPC of the specified region and creates a new public subnet for the EC2 instance. This error would suggest that the <IP_ADDRESS>/16 subnet overlaps with an existing subnet within that VPC in the selected region.
The cidr range is configurable through the variable vpc_subnet_cidr_block to override the default value. It's hard to reccommend what to set this to without knowing what other subnets are configured in your VPC, but try setting this to <IP_ADDRESS>/16 as this should work with the default VPC configuration.


Assign values to Error enums to avoid ordering issues

As enums have an underlying value that change according the order in the declaration, we can prevent these issues by assigning a value to each.
For example, Substrate does this:
pub enum ErrorCode {
    /// Returned when BEEFY RPC endpoint is not ready.
    NotReady = 1,
    /// Returned on BEEFY RPC background task failure.
    TaskFailure = 2,
}

Implementation of the types and their (de-)serialisation via SCALE codec.
https://docs.substrate.io/reference/scale-codec/
<EMAIL_ADDRESS>

also blocked by #645

After many questions we figured out that we do not yet need to do this.
Why not?

Metadata does contain the enum name connection to the enum. So people pulling the metadata don't care if we switched things around

Why maybe eventually?

People who are not relying on fetching the metadata and processing it at runtime would potentially misinterpret the data.

Current:

After release we should try hard not to re-order so that the changes to the metadata are limited to try to prevent churn, but we will not be offering any guarantees to maintain the encoding enums.

Future:

Perhaps if we remove an older enum value, we would want to number the remaining ones to prevent churn, but still no guarantees


Fixes heat featureflag check always returning false

@javisantana
I was putting renderTabs in the wrong class :sweat:

:es:

need to check that the user property exists to pass the tests though

Frontend tests were OK :+1: (details)


Add reminder to update side effects documentation for 'compatible'

The side effects table of 'compatible' in runtime/doc/options.txt got
quite out of sync and was just updated. Hopefully this comment will
reduce the likelihood of reoccurrance.

Hmm, I don't think that comment is read when adding an option.  Not useful.
Better have some way to generate the help table from the options.

For future googlers, I wrote a script to update the 'compatible' table for an input file formatted like this


attribute typo

I've spotted a typo in the proto: it should be latestWkid instead of lastestWkid

This would be resolved by PR #9

Making an upstream issue for this


nixos/maddy: Better description, user and group handling

Motivation for this change
This PR tries to fix several problems mentioned by @aanderse in an earlier PR https://github.com/NixOS/nixpkgs/pull/137260 , especially being more precise about the default configuration used by the maddy module.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

I'm struggling to squash the commits since they include merge commits :(

I'm struggling to squash the commits since they include merge commits :(

You should do rebase instead. I would suggest rewriting your history at this point. Drop the merge commit, iterate through the history and fixup as needed?
If you're having issues and want to work through this you could hop on discord or matrix and find someone to walk through it with you. If you catch me at the right time I'd be happy to.

Hey, thank you for this! I was experimenting with MTAs and this came in really handy.
I'm not sure this is in scope for this PR, but may I also suggest adding something like restartTriggers = [ config.environment.etc."maddy/maddy.conf".source ]; so the service is restarted whenever the configuration changes?

Hey, thank you for this! I was experimenting with MTAs and this came in really handy.
I'm not sure this is in scope for this PR, but may I also suggest adding something like restartTriggers = [ config.environment.etc."maddy/maddy.conf".source ]; so the service is restarted whenever the configuration changes? Unless, of course, this is not in scope or if someone would say this is not a good idea.

Doesn't the service restart automatically with nixos-rebuild in case something has changed?

Good catch @drdaeman!
@onny no because the service doesn't change.

Okay added it :)

Yep works fine for me here

Yep works fine for me here

Thanks again @onny! ❤

Thanks again @onny! ❤


Where can I find the remaining 50k dataset in COCONut-L?

Thank you for this outstanding work.
I have noticed that the number of annotations released by COCONut-L currently includes COCO trains, COCO unlabelled，60k objects365， Roughly around 300k, but declared as 358k in the paper and github?

At the same time, I found that the images in object365_train_cnseg.tar https://huggingface.co/datasets/xdeng77/coconut_large/tree/main
do not correspond to the subset of images you provided for objects365 https://drive.google.com/drive/folders/1ZumOrkikulK96MbBxayO07JP_HV-fgTW

now it is available at huggingface, should update kaggle soon.


Fix deep matplotlib waveform plot

@tunnel reported that plotting large time ranges (> few ms) with st.plot_waveform(..., deep=True) can result in an empty records plot.

I suspect the cause is that the records matrix became very large (many GBs), causing rendering goofs.
This modifies st.records_matrix (used by the waveform plot) to create downsampled matrices if the requested time axis exceeds 20_000 samples (configurable through the max_samples argument). This solves the problem:
import straxen
st = straxen.contexts.xenon1t_dali()
run_id = '170204_1710'
df = st.get_array(run_id, "event_info")
event = df[4]
st.plot_waveform(run_id, time_within=event, deep=True)

It also has other benefits:

We actually fill a smaller matrix with downsampled waveforms, rather than downsampling a huge matrix. Thus preparing the plot (not just the rendering itself) is much faster and takes far less memory.
You can now plot very large time ranges with the matplotlib display -- though the color scale inference could use improvement -- e.g.

st.plot_waveform('180215_1029', seconds_range=(0, 10), deep=True)

Hi, this is just a bug fix, so we can merge now.


ColourGroup should be ColorGroup

The naming of ColourGroup is inconsistent.

ColourShade also needs changing

Hmm, aren't these both already ColorGroup and ColorShade? :smile:


Fix ajax endpoint url, fixes #240

This pull request fixes #240
What's Included in This Pull Request
WooCommerce 3.2.0 introduced different handling of Ajax endpoint
URLs, including call to home_url() function, for more info see
woocommerce/woocommerce#16991
This breaks Polylang language detection if language is set from the
directory name in pretty permalinks.
To address this issue we need to add WooCommerce class-wc-ajax.php
file to the white list of the Polylang home_url filter, enabling
home_url() function to return correct URL according to the language
set.

Thank you for valuable contribution :smile: , This issue has been on my todo list for a while now, but I could not find the time for it


Added monitor_overview.md

A diagram of the workflow will be added later.

This PR will close issue #924 when merged.


OperatorFramework Controller tests are flaky

Many tests in /src/OperatorFramework/test/UnitTests/Controller are heavily timing-dependant and flaky.
We've disabled the most noisy offenders for now:

DelayingQueueTests

DelayingQueueAddsWhenTimePasses
NoAddingAfterShutdown

LimiterTests.cs

WaitAsyncCausesPauseLikeReserve
ManyWaitsStackUp

Is operator framework going to be used by the new k8s implementation?

#1642 removes the unused parts of the OperatorFramework.


Create project=johhny

Bot. Do NOT merge this! See https://github.com/IBM/watson-speech-translator/pull/36 for more info.


Change docs so that the decorator hook is run for builds in general (sdist and/or wheel)

Change docs so that the decorator hook is run for builds in general (sdist and/or wheel) and not just wheel:
from
[tool.hatch.build.targets.wheel.hooks.decorators]
to
[tool.hatch.build.hooks.decorators]
See: https://hatch.pypa.io/1.9/config/build/#order-of-execution

Incorporated in #13


fix: code format

Signed-off-by: fengshunli<EMAIL_ADDRESS>Description
Related issues

Close #XXX

Remove this section if you don't have related PRs.
Checklist

[ ] I've read the guidelines for contributing to this repository.
[ ] I've added tests that prove my fix is effective or that my feature works.
[ ] I've updated the documentation with the relevant information (if needed).
[ ] I've added usage information (if the PR introduces new options)
[ ] I've included a "before" and "after" example to the description (if the PR is a user interface change).

@fengshunli Please open an issue for code format , every PR must have issue for it

updated


Added a general mechanism to let cells ask confirmation to the app before doing something

The first usage is the click on a link in the chat history to intercept universal links.

LGTM - the added mechanism is quite complex.


Add Thanos Rule

This PR,

upgrades dependencies
adds/modifies manifests to add Thanos Rule

cc @squat

Is there anything left to do for this one? 😊

@metalmatze If you give your blessing, I'll merge it :)

Rebased after template splits.

cc @metalmatze Could you have another look at it?

The component should probably be named Ruler when we first merge and introduce this already. :blush:

Maybe we should do the renaming, as another PR when upstream proposal also happens. Right now, thanos and kube-thanos refer it as thanos rule. At least I have to change it on kube-thanos first.
So maybe we can merge this one since this also has pvc/volume fixes in it.

@metalmatze To fix volume issues, this disables https://github.com/thanos-io/kube-thanos/pull/76 hiding the volumes.

I think this was the right move: https://github.com/observatorium/configuration/pull/122#issuecomment-565504394
otherwise it's quite funny to see half thanos-rule and half-thanos-ruler. It seems like a label mismatch just waiting to happen.

@metalmatze @squat  Changes in https://github.com/thanos-io/kube-thanos/pull/77 are also reflected now.


this is artifacts it's normal when you're training gans

like this

@WYanNuo Hello，I also encountered a similar problem. Have you solved it？


Add CameraScriptsLoader.rbxm

This makes it easier for users to install the injector by allowing them to drag/drop the file onto an open Studio window, solving #1 .

Added a release


UX: moves new channel button to browse view

Removes the dropdown with browse/create from the sidebar. As a side effect:

it's now a button using the pencil icon, leading to browse page
the new channel button is at the top of browse page

It's only changing behavior for staff, regular users were already not seeing the dropdown as they can’t create channels.


bug: Footer toolbar shouldn't use --ion-safe-area-bottom when tabs are below

Bug Report
Ionic version:

[x] 4.x
Current behavior:

iPhone X/Xs....:
The footer inside of a toolbar has following line CSS:
padding-bottom: var(--ion-safe-area-bottom,0);
Sadly this padding is applied even there is a tab-bar under footer. In my example the calculated padding-bottom equals 34px.
Example image:

Expected behavior:
If there is a tab-bar under the footer, the padding-bottom should only be applied on the tab-bar.
Steps to reproduce:
Take any Ionic tab-bottom-project. Insert a footer with a toolbar inside and execute this on e.g. IPhone Xs.
Other information:
I found a related thread about this topic for an older version of Ionic: https://github.com/ionic-team/ionic/issues/13615
Ionic info:

Ionic:

   Ionic CLI                     : 5.2.4 (/usr/local/lib/node_modules/ionic)
   Ionic Framework               : @ionic/angular 4.7.1
   @angular-devkit/build-angular : 0.13.8
   @angular-devkit/schematics    : 7.2.4
   @angular/cli                  : 7.2.4
   @ionic/angular-toolkit        : 1.4.1

Capacitor:

   Capacitor CLI   : 1.1.1
   @capacitor/core : 1.1.1

Utility:

   cordova-res : 0.6.0
   native-run  : 0.2.8

System:

   NodeJS : v12.6.0 (/usr/local/Cellar/node/12.6.0/bin/node)
   npm    : 6.9.0
   OS     : macOS Mojave

There’s already a ticket for it here (and a hack to work around it): https://github.com/ionic-team/ionic/issues/17560

Okay thanks. I will close this issue.


Line 74 Group Error

Stops installing with this error. No idea what to do.

Traceback (most recent call last):
File "install.py", line 74, in
trello_grocery_board = m.group(1)
AttributeError: 'NoneType' object has no attribute 'group'

Not enough info to go on here, sorry.


robotcar-season eval run very slowly

Hi, thanks for your excellent work.
I finished robotcar dataset preparation and can successfully run eval program. However, the process takes too much time (around 2000+ hours estimation for matching 4.7M+ data). Is it normal? Or any mistakes maybe happen?
Also, does skip-match work? And can I skip that? Could you help me and explain?
Thanks for your time!

Hi! I'm facing the similar problem, have you solved it?

Hi @XJTU-Haolin ,
I think the easiest way to check this is by estimating how long does your target matching method take to match a single pair. And then you can compute to how long it requires to eval 4.7M+ data.
To quickly evaluate the runtime of your matcher within your environment, you can use the example pairs or hpatches. And make sure you are using GPU when you run the matcher. It should be using that by default as long as it is visible in your environment.

Hi @XJTU-Haolin ,
I think the easiest way to check this is by estimating how long your target matching method takes to match a single pair. And then you can compute how long it requires to eval 4.7M+ data. To quickly evaluate the runtime of your matcher within your environment, you can use the example pairs or hpatch sequences. And make sure you are using GPU when you run the matcher. It should be using a GPU by default as long as it is visible in your environment.

Thanks for your excellent repo and nice response. I will find time to check again.


Add files via upload

As I am participating in the Hacktoberfest 2022, please add the label of hacktoberfest to it so that it could be helpful for me.


Support pushing to blender api endpoint

I created a PR a while ago for editor trails that would push trails to blender. I know skyslide would appreciate it if you supported the format we developed (it's live in blendermania atm). Happy to help with porting.
to find relevant code in the diff, ctrl+f for Setting_UploadTrailsURL

Thanks for the link. I'll have a look when I look at JSON import/export to support your format.


Wrong range selected in the datepicker

This is a (multiple allowed):

[x] bug

Framework7 Version: 1.5.3

Platform and Target: Material theme & iOS Theme.

What you did
Select a date from range datepicker
Expected Behavior
When you select the range, all days shoud be selected in the calendar.
Actual Behavior
The last day on some cases is not marked, but the date above is correct.
This issue is reproducible in the kitchen sink, both material theme and iOS theme.
The gif shows some examples with the wrong behavior, and the last one is the expected behaviour.

Well, for some reason i can not replicate it and it works fine for me. Can anyone confirm there is such issue cc @valnub @ZanderBrown

I can't replicate it in the online Kitchen Sink demo at http://framework7.io/kitchen-sink-material/ using Google Chrome on latest macOS.
@tatxini Which browser and OS are you using?

Yes, I can replicate it using Google Chrome on macOS.
The problem only occurs if you select exactly the same date range as in the gif (March 16th-30th).
On other date it works correctly.

I also see another problem in the date picker. When I click on a day, it will not always select the day (nothing happens). Sometimes I have to click multiple times on a day and then it will select it. I remember that someone posted about a change in Google Chrome that breaks clicking behavior in F7. Don't remember that post tho. This problem does not exist in other browsers, e.g. Firefox. Other users also reported similar issues with components not always reacting when clicked.

I can reproduce it as well but I haven't discovered what the magic combination is. The first few date ranges I select work fine then eventually the date range stops highlighting the last day.

Still works fine for me 😞 What is environment. I'm checking at:
http://framework7.io/kitchen-sink-material/
macOS 10.12.3
Chrome OSX 56.0.2924.87 (64-bit) (with both emulating mobile device (android) and not)

Was about to post about this issue, Happens for me on windows 7 64bits, chrome 56.0.2924.87 and on firefox 51.0.1. Also I can reproduce it on iPad pro v10.2 & a Samsung Note 5 running android 6.0.1

Try a range from 15 to 31 march 2016 or october 2016.

http://framework7.io/kitchen-sink-material/
Chrome 56.0.2924.87 (64-bit)
macOS 10.12.3
No emulator

Make sure to follow the steps exactly:

Open kitchen sink (Material)
Select Calendar
Scroll down to select range
Click on 15
Click on 30

This seems to affect march and october only

I think I found the cause: whenever daylight saving dates are in the selected range the issue happens.

😂 That would be funky but sounds plausible since Vladi is in Russia. Do you have daylight saving time over there?

Followed @valnub 's steps. Can confirm bug on: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3013.3 Safari/537.36
Might be something in the daylight saving theory, the UK changes from GMT (+0000) to BST (+0100) on the 26th which is, of course, between the 15th and 16th of March

OMG! Another thing we can blame on daylight savings time.

Wow, that is why it works fine for me

Ok, i believe we already fixed it by @Tarekajaj PR

www.3aiqi.com


Update indice_uv.md

:tada: This PR is included in version 1.24.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


TestRequestSFR

Lorem Ipsum - TestRequest - DevOps FFM

Closing this issue as this appears to be a test. Please see the repository contribution guidelines.


DOC: some odr models are missing docs

As discussed in https://github.com/scipy/scipy/pull/12052#discussion_r422497598, some odr models missing docs.
Only odr.polynomial has doc, because it is a function with docstring, however, other models: exponential, multilinear, unilinear, quadratic does not have doc, because these are instances.
Due to this issue, it is difficult for users to know how to use these models.

My possible solution is to change these 4 instances to each factory function with docstring like polynomial. Code consistency would be better.

This goes all the way back to the creation of the odr module 2005, in commit 3f92506da. The reason for the difference will be that polynomial takes a parameter (order), but the other ones are formulas with no parameters. Still, this looks a little weird.
Maybe the right fix here is to change these four Model instances to be instances of Model subclasses, that can then have their own docstring and doc page in the html docs. This should be a change that's backwards compatible enough.
@rkern any thoughts?

Yeah, that's a plausible workaround.


ggml-python server

I just wanted to express my gratitude for your effort on this project. I'm a fan of llama-cpp-python and the prospect of also running mpt, replit, and starcoder models is incredibly exciting, especially given their permissive licensing.

I just realized that the focus of this repo is decidedly more low-level. Is it then correct to assume that the most efficient way to run llama-cpp-python with non-llama models, such as mpt, replit, or starcoder, would simply involve setting the environment variable LLAMA_CPP_LIB to the corresponding binary, given that most arguments appear to align?
In that case it might already be possible to use starcoder with gpt-code-ui.

Hey @yadamonk thank you so much!
To clarify a little, I have two goals for this project:

Expose the ggml.h interface to use directly from python. This is currently possible but as you've probably realised quite low-level.
Expose the ggml-based models in a way that makes them easy to install and use in the same way as the llama-cpp-python project.

The issue with the second goal is that those models do not yet have APIs in the same way that llama.cpp does. This leaves two options for implementing these, first is to adapt those c++ examples into unofficial APIs and bind to those or write python implementations using the low-level or one abstraction layer above. I'm going to try the second approach as it seems more flexible.

Hi @abetlen, thank you so much for clarifying. I agree that the second approach is the right way to go about things long term.

I'm super excited to see that you're working on an example for replit, especially given the recent code instruct version by teknium. I know it's still early days but do you have a rough estimate when it might be possible to connect the replit model to a web UI?

@yadamonk tonight hopefully!
Also, I've tested it with Teknium's CodeInstruct v2 model and it works great, q4_0 should be able to run with ~2GB memory required.

Quick preview of what I'm working on, this is using gpu weight offloading with approx 1.5GB of VRAM. With the fastapi server I was able to drop it into vscode (and vim) and get a local copilot working
https://github.com/abetlen/ggml-python/assets/6826477/bcf4be54-6f5a-49e3-963f-61c0b7c96d9e

Absolutely amazing! I hope I'll find some time to play with it today. Thanks again for all your work moving the ecosystem around ggml forward.


Output 'true' node path in report

Given a custom rule such as
 common-responses-server-error:
  description: "Missing 500 response"
  severity: error
  category:
    id: operations
  given: $.paths.*.*
  then:
    field: responses.500
    function: truthy

When the report is output to the terminal, each infraction is listed correctly, however the path shown in the output is just the verbatim given from the rule - this makes it difficult to quickly grok which operation is the issue.
Line / Column | Severity | Message                                           | Path
(23:7)        | error    | Missing 500 response: `responses.500` must be set | $.paths.*.*
(34:7)        | error    | Missing 500 response: `responses.500` must be set | $.paths.*.*
(45:7)        | error    | Missing 500 response: `responses.500` must be set | $.paths.*.*
(56:7)        | error    | Missing 500 response: `responses.500` must be set | $.paths.*.*
(78:7)        | error    | Missing 500 response: `responses.500` must be set | $.paths.*.*
(100:7)       | error    | Missing 500 response: `responses.500` must be set | $.paths.*.*

Is it possible to have the output show the 'true' path to the offending node? (e.g. $.paths./foo.get instead of $.paths.*.*)

Hi,
Unfortunately there isn't currently a way for vacuum to know the 'true' path when using some core functions, the way the engine works is the nodes are extracted according to the json path lookup, and then evaluated, the path provided is the path the lookup happens on.
Once those nodes come in to the function, they are evaluated.
I think some better docs by help explain this concept a bit better.
Trying to capture the path of the evaluated could be added as a feature request however.

@daveshanley  This is very much necessary for my use-case - spectral does this so its a feature gap that is preventing me from switching over to Vacuum

I am in the process of upgrading all the built in rules, to ensure paths are correct. This is being powered by a new product I am developing.
This use case however cannot be satisfied yet however, even with a much more powerful model underneath that knows the path to every single object, custom rules do NOT use this model (it's programatic).
This means that there is no way to know where in time and space the result is coming from, because custom functions use the JSON Path provided by the rule.
Custom rules have no concept of paths, because the code extracts raw nodes from the underlying document, based on the query provided.
The only way to extract the paths correctly, is to use the datamodel programmatically.
The only way to make this feature possible with custom rules, is to build a custom JSON Path parser and crawl the model
looking for those nodes. This is a huge undertaking and well out of scope for the time I have spare for free work.

@daveshanley Understood. I wonder how Spectral achieves this (maybe this is why they are so much slower haha)

wondering given the fact that you have the line number if this could not be then back tracked ?
ie going from the leaf and get back to the root (and so the path ) ...

@daveshanley Understood. I wonder how Spectral achieves this (maybe this is why they are so much slower haha)

The library they have may or may not allow for reverse yaml traversal (you can walk backwards from any point in the AST). However the one we use does not, it's a one way walk down only, there is no parent reference to allow us to walk back up the tree.
https://pkg.go.dev/gopkg.in/yaml.v3#Node
There is no Parent property available for us to walk back up.
Once we drop down into a node after the search using the path.. we can only build from that point down, we have no idea where we came from. It's a one way journey.
The solution would be to update the library to add a Parent property to allow us to walk backward. We have done that with our programatic model for OpenAPI in libopenapi which is why the OWASP rules all now have accurate paths.
But, for this, we have no way forward, without modifying the underlying yaml library to support a two way walk.

wondering given the fact that you have the line number if this could not be then back tracked ? ie going from the leaf and get back to the root (and so the path ) ...

As mentioned in the above comment, we can only walk one way, it's a one way tree, you cannot walk back up it.
https://pkg.go.dev/gopkg.in/yaml.v3#Node

@daveshanley Understood. I wonder how Spectral achieves this (maybe this is why they are so much slower haha)

The library they have may or may not allow for reverse yaml traversal (you can walk backwards from any point in the AST). However the one we use does not, it's a one way walk down only, there is no parent reference to allow us to walk back up the tree.
https://pkg.go.dev/gopkg.in/yaml.v3#Node
There is no Parent property available for us to walk back up.
Once we drop down into a node after the search using the path.. we can only build from that point down, we have no idea where we came from. It's a one way journey.
The solution would be to update the library to add a Parent property to allow us to walk backward. We have done that with our programatic model for OpenAPI in libopenapi which is why the OWASP rules all now have accurate paths.
But, for this, we have no way forward, without modifying the underlying yaml library to support a two way walk.

Yeah that makes sense. Would be nice to have eventually, but I understand this would probably be a big undertaking and is out of scope

There is a solution to this, inbound.

There is a solution to this, inbound.

OMG amazing!

Fixed in v0.12 https://github.com/daveshanley/vacuum/releases/tag/v0.12.0


CloudFormation autoscaling group PropagateAtLaunch not setting properly

When creating a CloudFormation autoscaling group, PropagateAtLaunch for tags is being reported as 'False' even though it was set to 'True' in the template

I can also work on this

Upon further investigation it's actually the ecs service directly that's not returning the correct response, the moto backend has the correct response, but the value is getting lost somewhere in the interaction with boto3

This is available as of moto >= 1.3.15.dev871


Allow #[derive(Data)] on empty structs

Detect structs with no fields and generate no-op
compare function accordingly.

Useful while iterating. The compile errors are quite off-putting otherwise.

yea this was an open bug, #246. Thanks!

Oh, I didn't even see the bug. In that case:
Fixes #246

I'd love to see tests for this; they can go anywhere in druid-derive/tests. The following should all compile:
#[derive(Clone, Data)]
struct Empty ();

#[derive(Clone, Data)]
struct Empty2 {}

#[derive(Clone, Data)]
enum Items {
One,
Two(),
Three{},
}

I actually was about to suggest that.
Do you have a preference for where to put lens tests? src/lens/test_derive.rs?

Done

Ah no I wasn't done yet, still some work to do on enums.

haha sorry. looked like a good checkpoint in any case, so a follow up sounds good.


Size of AST does not seem to contribute to costs

Summary
See https://github.com/input-output-hk/plutus/blob/master/plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExBudget.hs#L57
@TotallyNotChase tried using the UPLC term λredm. λctx. λx. <huge> as a minting policy (before Vasil), and it the costs didn't seem to change compared to a small term.
Steps to reproduce the behavior
Try evaluating term.
Actual Result
Size doesn't affect costs.
Expected Result
They should according to the comments, but maybe the comments are outdated?
Describe the approach you would take to fix this
No response
System info
_

In particular, I tried with a "always succeeds" minting policy, where the I alternated the body between constants and lambdas (making sure no extra evaluation takes place). Whether the script was (\redm -> \ctx -> <huge constant>) or (\redm -> \ctx -> \i -> <huge body>) - the CPU/Steps and Mem cost reported on cardanoscan's "Contracts' tab was always the same- 800 Mem and 208511 CPU - which does indeed account for the CEK machine costs (for each term, and a startup cost) and the builtin cost model (though there are no builtin function calls in this case). There is no extra cost depending on the actual size of the AST (e.g constants, no matter the size, yield the same cost).
This does align with my understanding of the cost model, but the comment linked seems misleading in that it suggests there is another category of costing - one that depends on the size of the AST.

Yes, the comment is wrong, the size doesn't contribute to costs.

Maybe it should?

Users already pay for AST size because it contributes to transaction size

Well, not with reference scripts, and it seems like rewarding smaller scripts might be logical.

That is a good point about reference scripts.

We've considered implementing aggressive inlining for Helios when generating UPLC, if however the size of the reference script gets an associated cost at some point in the future this optimization won't be worth it. @michaelpj do you think that this could indeed change?

Unclear. We might have to do it for correctness, I don't know.

Maybe @dcoutts has some thoughts? Currently the incentives are such that you make almost 16 kiB scripts, but I'm not sure how the penalty of needing more inputs above that affects it, and what is best for Cardano, i.e. what aligns fees and costs the mosts.

@michaelpj

That is a good point about reference scripts.

Unclear. We might have to do it for correctness, I don't know.

So what about that? This seems like an important issue.
@JaredCorduan maybe you have some insight?

I'm not very familiar with how the evaluator counts the execution units, but I do know that the script context is only given a hash of each reference script, so it is not too surprising to me that the costs are blind to the size (besides the initial cost of creating the reference script). I am not aware of any related correctness issues.

To be clear, here's the potential issue:

Create a script S, that is very large but evaluates very quickly, e.g. if false then <lots of stuff> else ().
Create N outputs locked by S
Spend those outputs using a reference script for S.
We do O(size of S) work in e.g. deserializing the script, but the user pays O(N) for evaluation and O(N) for tx size.

I don't think this is a big problem, but it was previously the case that they would pay O(size of S) due to tx size fees.

We do O(size of S) work in e.g. deserializing the script, but the user pays O(N) for evaluation and O(N) for tx size.

This doesn't sound very good. If I'm reading it correctly (not sure about that), it means we can do more work than what we charge for, which is nasty.
It's been half a year, do you know if we did any work on this front or if there was any progress at all? Do you have an opinion on whether we should prioritize this issue?
Also CC'ing @zliu41.

Nothing beyond my previous comment.

I still don't think this is too bad. I'm unsure if it's something we could retrofit in, either.

The reference scripts issue is fixed in the Conway era, hence this issue can be finally put to rest, since for other types of scripts it's all fine as per Michael's comment:

Users already pay for AST size because it contributes to transaction size


request to add Maricopa Community College

@FreddyKilo regrettably, this request cannot be satisfied. Please see #2393 for the details.

No worries, was wanting to try DataGrip, guess I'll have to wait until I transfer to ASU, see you in many months 😄

@FreddyKilo If you wish to try it, you can use 30-days free trial version. You can also consider applying with ISIC card or another document which proves that you are a student. There are several options available at jetbrains.com/student. Please let me know if I can help more. Thank you!

Thanks for offering the options :)


[SPARK-15072][SQL][REPL][EXAMPLES] Remove SparkSession.withHiveSupport

What changes were proposed in this pull request?
Removing the withHiveSupport method of SparkSession, instead use enableHiveSupport
How was this patch tested?
ran tests locally

Test build #57569 has started for PR 12851 at commit 7a07671.

LGTM

Test build #57569 has finished for PR 12851 at commit 7a07671.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57569/
Test FAILed.

Test build #2961 has started for PR 12851 at commit 7a07671.

Test build #2961 has finished for PR 12851 at commit 7a07671.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Seems like some legitimate failure/?

Jenkins retest this please.

Test build #57656 has started for PR 12851 at commit 7a07671.

Test build #57656 has finished for PR 12851 at commit 7a07671.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57656/
Test FAILed.

@rxin yupp failed tests are in sql hive tests, waiting for https://github.com/apache/spark/pull/12890

Test build #57812 has started for PR 12851 at commit 17e0703.

@techaddict a general point is to try to get the SparkContext from the SparkSession rather than the other way round. Right now we call getOrCreate() and assume there's a SparkContext nearby, which may be true but is kind of brittle and outright wrong in some cases.

Test build #57812 has finished for PR 12851 at commit 17e0703.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57812/
Test FAILed.

Test build #57849 has started for PR 12851 at commit 7ad494d.

Test build #57849 has finished for PR 12851 at commit 7ad494d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57849/
Test PASSed.

@rxin @andrewor14 Comments Addressed, now all tests are passing 🎉

@techaddict this is getting really close. For the regression test for SPARK-8489 I would just fix it later since this patch doesn't rebuild that jar anyway.

Test build #57904 has started for PR 12851 at commit 5e464ad.

LGTM

Test build #57904 has finished for PR 12851 at commit 5e464ad.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57904/
Test PASSed.

Merging into master 2.0


Unable to log in with GitHub Desktop client present

The more we know about your system and use case, the more easily and likely we can help.
Environment info

OS: Win10 Enterprise (1607)

Description of the problem:
Unable to log in. I suspect this has something to do with the fact I have the GitHub Desktop client running on the same machine and in that one I'm logged into our enterprise GitHub server while Lepton attempts to log into the public GitHub. (this is just a suspicion though).
Logs
Logs sent to<EMAIL_ADDRESS>Sent the Lepton logs from today + Fiddler capture from the login attempt

Try delete C:\Users<username>\AppData\Roaming\Lepton\ .

That did it!


No values in the Solcast entities

The problem
Hello everyone
Hopefully I'm asking my question in the right place.
I have successfully downloaded and installed the Solcast integration. A corresponding API query automation is running in the background and the solcast.json is present. Unfortunately, I am not getting any values in the Solcast service entities.
Reloading the Solcat integration several times has not been successful so far.
Do you have any idea where I could start to fix this?

Thank you very much for your help
What version of Solcast Integration has the issue?
v4.0.35
What version of Home Assistant are you using?
No response
Diagnostics information
No response
Logs debug/info/error output
2024-07-09 11:57:38.070 INFO (MainThread) [custom_components.solcast_solar] SOLCAST - Service call: update_forecasts
2024-07-09 11:57:38.072 ERROR (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - http_data error: Traceback (most recent call last):
  File "/config/custom_components/solcast_solar/solcastapi.py", line 1013, in buildforecastdata
    self._forecasts_start_idx = self.calcForecastStartIndex()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/solcast_solar/solcastapi.py", line 1031, in calcForecastStartIndex
    _LOGGER.debug("SOLCAST - calcForecastStartIndex midnight_utc %s, idx %s, len %s", midnight_utc, idx, len(self._data_forecasts))
                                                                                ^^^
UnboundLocalError: cannot access local variable 'idx' where it is not associated with a value

2024-07-09 11:57:38.077 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.077 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 0 st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.078 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_day st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.078 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_time_day st 2024-07-08 22:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 1 st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_day st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.079 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_peak_w_time_day st 2024-07-09 22:00:00 end 2024-07-10 22:00:00 st_i 0 end_i 0 res None
2024-07-09 11:57:38.080 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-10 22:00:00 end 2024-07-11 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.080 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 2 st 2024-07-10 22:00:00 end 2024-07-11 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.081 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-11 22:00:00 end 2024-07-12 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.081 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 3 st 2024-07-11 22:00:00 end 2024-07-12 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.081 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-12 22:00:00 end 2024-07-13 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 4 st 2024-07-12 22:00:00 end 2024-07-13 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-13 22:00:00 end 2024-07-14 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 5 st 2024-07-13 22:00:00 end 2024-07-14 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:57:38.082 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_total_kwh_forecast_day st 2024-07-14 22:00:00 end 2024-07-15 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 11:57:38.084 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_day 6 st 2024-07-14 22:00:00 end 2024-07-15 22:00:00 st_i 0 end_i 0 h.len 0
2024-07-09 11:58:00.158 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'cloud_coverage' when rendering '{{ tagesvorhersage['weather.forecast_home'].forecast[1].cloud_coverage }}'
2024-07-09 11:59:00.157 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'cloud_coverage' when rendering '{{ tagesvorhersage['weather.forecast_home'].forecast[1].cloud_coverage }}'
2024-07-09 12:00:00.156 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'cloud_coverage' when rendering '{{ tagesvorhersage['weather.forecast_home'].forecast[1].cloud_coverage }}'
2024-07-09 12:00:00.193 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 10:00:00 end 2024-07-09 11:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.194 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_remaining_today st 2024-07-09 10:00:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.194 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 11:00:00 end 2024-07-09 12:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_custom_hours st 2024-07-09 10:00:00 end 2024-07-09 11:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 09:50:00 end 2024-07-09 10:10:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:20:00 end 2024-07-09 10:40:00 st_i 0 end_i 0 res 0
2024-07-09 12:00:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:50:00 end 2024-07-09 11:10:00 st_i 0 end_i 0 res 0

2024-07-09 12:05:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 10:00:00 end 2024-07-09 11:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.195 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_remaining_today st 2024-07-09 10:05:00 end 2024-07-09 22:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.196 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_n_hour st 2024-07-09 11:00:00 end 2024-07-09 12:00:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.196 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_forecast_custom_hours st 2024-07-09 10:05:00 end 2024-07-09 11:05:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.196 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 09:55:00 end 2024-07-09 10:15:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.197 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:25:00 end 2024-07-09 10:45:00 st_i 0 end_i 0 res 0
2024-07-09 12:05:00.197 DEBUG (MainThread) [custom_components.solcast_solar.solcastapi] SOLCAST - get_power_n_mins st 2024-07-09 10:55:00 end 2024-07-09 11:15:00 st_i 0 end_i 0 res 0

Additional information
No response

What's in the solcast.json file? A debug log if the integration starting up would be handy.
That error message is pretty rough under the circumstances, and wants cleaning up, so sorry about that.
Also, what's in solcast-sites.json file? (Please redact your latitude/longitide.)

The error 'symptom' is because the forecasts have not been able to be built, so it follows that there is no data from Solcast.

The service call to update occurs, as noted.
This then calls http_data(), which iterates through the sites. If there is no failure then buildforecastdata() is called.

The only way for this to happen and return that unfortunate error is if there are no Solcast sites to iterate. (I need to fix this, because no Solcast sites should then not attempt to build a forecast, and this should clearly be spelled out...)
So we need to know what is in solcast-sites.json (GPS location redacted if it's in there), so need a log of the initial start of the integration.
On startup, the sites are retrieved. This can sometimes be initially problematic because of the Solcast API responding with "429/Too busy" for hobbyists, hence the need to repetitively restart if a solcast-sites.json has not been created (which is a cache, to expect future "429/Too busy" failures).

solcast.json:
{"siteinfo": {}, "last_updated": "2024-07-09T11:32:17.010580+00:00", "version": 4}
solcast-sites.json file does not exist
home-assistant.log.txt

This has exposed an interesting issue, by the way.
Should there be no sites in the cache, and the sites fail to load, then on the next call to update the forecast (in the present version) the solcast.json will be written, as possibly an empty file containing {}. This is "possibly" empty because there is an initial call to load this file into memory, so writing it should cause no harm. But if the load call fails too for some reason?
I have already generated a fix for the attempt to build a forecast where no sites have been attempted, so the "idx" error shoud not recurr in 4.0.next.

There's the problem. A connection timeout. You just need to work out why your HA cannot access solcast.com, which if that were the case would result in a timeout. Solcast are usually pretty quick to connect, even if the conversation ends abruptly with a 429/Too busy, and that's not occurring in this case.
The integration must have the site detail to go get forecasts.
SOLCAST - sites_data get sites timed out, attempting to continue
SOLCAST - No cached site(s) are available for ******EBBmrt to cope with Solcast API timeout
SOLCAST - sites_data Timed out async wait for connection to solcast server, and one or more site caches failed to load. This is critical, and the integration cannot function reliably. Suggestion: Attempt integration reload.
SOLCAST - sites_usage Connection Error - Timed out connection to solcast server
SOLCAST - load_saved_data site count is zero. The get sites must have failed.

How do I check or provide access from HA to solcast.com?
How do I provide the location details to the integration?

By the way -> thank you very much for the answers to my questions so far.

I get an answer back in the way described above.

Just to confirm, this curl was performed on the server that is running Home Assistant, yes?

Is this HAOS?
Is this a Docker set-up?
Is this manhandled into Kubernetes?
Anything else about the environment that is 'interesting'?

All I know is that it is integration v4.0.35, and you have a networking issue that no one else has, and we do not have an integration issue that I can see.
Right now I have absolutely nothing to go on. The call works, so it should work from the integration, hence should write the solcast-sites.json, and then move on to a forecast get.

Yes, this curl was executed on the server running Home Assistant.
Core 2024.7.1
Supervisor 2024.06.2
Operating System 12.4
runs on an intel NUC

Thanks for that. That's exactly the same as I'm running, except Proxmox VM for me.
I'm really stumped here, @pagasam, and can't think what could be causing a failure to connect to api.solcast.com.au, especially as a curl from HAOS works fine.
The logged timeout is genuine, 60 seconds in both cases, so it's not like the code is prematurely declaring a timeout, and I assume that the curl returned near instantaneously.
There is a beta upgrade available to v4.0.37, but I honestly don't think that's going to make any difference. That release focuses on some new attributes, plus improvements to exception handling. You could try a re-download/restart, but I can't see how it would help.

I wonder if this could possibly somehow be IP v6 related. If DNS returns a v6 address to use then your v6 config in your router could be weird. DNS may have returned v4 for the curl.
You could try turning v6 off in the HA network configuration.

You were right. It was the v6 configuration. I have now deactivated this and only use v4. The integration now runs smoothly with the available beta upgrade to v4.0.37.
Thank you very much for your help, I really appreciate it

Happy to be an Agony Aunt for non-related issues, @pagasam...
Do please enjoy. And especially enjoy v4.0.37. We're building an even greater integration, with more activity, and more thought, commit by commit.


Update readme to allow for new users

To use the skupper-docker command you must first build a local container:

Make sure the following software is installed:

Docker
Golang

Use the following command to create a local skupper-docker command:

make build-cmd

You can now use skupper-docker, noting that where messages mention skupper, substitute skupper-docker

@ajssmith Does above help?

hi!
can I install skuper-docker to windows os?

@Aleks3050  I haven't tested much, but seems to work using WSL 2, see https://docs.docker.com/docker-for-windows/wsl/

the original issue was addressed in https://github.com/skupperproject/skupper-docker/pull/16


WIP Create locking mechanism for check file changed

Closes #1118
This pull request provides a proper solution to overcome the problem described #1118 on a windows machine.
The bug is internally caused by the fact that the mainWindow#focus event is triggered twice. Therefore the contentChanged-dialog is opened twice.
Except for other platforms, e.g. macOS, the dialogs on Windows have its own focus, and are not attached to the mainWindow. That means on the time the user clicks on some action of the contentChanged-dialog, the dialog's got closed and it will focus the mainWindow a second time. Surely we have a check that the dialog only opens when the file.lastModified has changed - which prevents us from more dialogs to be opened - but the file read operation right after the user interaction costs a small amount of time. This is enough space to open the dialog a second time right after with the old tab-file state.
One can illustrate the whole misleading workflow as follows:

make changes in an external text editor
switch to modeler
1st mainWindow focus event
change detected --> first dialog opened with a different focus
user chooses option --> dialog got closed --> updating
meanwhile 2nd mainWindow focus event --> second dialog got opened, just before the tab updating was successful from the first execution

To prevent this behavior, we have two options:

prevent dialogs having its own focus on Windows
create a lock which disallows opening a second dialog

I was not able to find a solution for the 1st option since it is natively dealt by Electron. The 2nd option is close to the solution we implemented regarding the namespace dialog inside the BpmnEditor. This is done by this PR (cf. 861a84b).
tldr
I think this locking pattern is a proper way to overcome the multiple focus problem regarding the contentChanged dialog since it helped us once before. After chatting with @barmac it comes clear that we might have to think about an overall locking functionality for dialogs to prevent this in future. This could lead to another issue following up.
Tasks to do

[x] provide test coverage

Tested on my Windows 8.1 VM

@barmac @nikku dd0344f3687f73ad07ded1f8a5e8b65e2d1e4dc1 now is introducing a flexible wrapping functionality for tab procedures, to make sure them only got executed once. Should be ready for review now.


Pure localization but still mapping

Hi all,
I running Pure localization with command:
roslaunch cartographer_ros demo_backpack_2d_localization.launch \    load_state_filename:=${HOME}/Downloads/b2-2016-04-05-14-44-52.bag.pbstream \    bag_filename:=${HOME}/Downloads/b2-2016-04-27-12-31-41.bag
But still mapping and merge with b2-2016-04-05-14-44-52.bag.pbstream map
Please help me, thank you

Hey, did you ever figure this out? I'm working on a project involving Cartographer and I'm running into the same issue. I have my lua config set up the same as what the documentation says (https://google-cartographer-ros.readthedocs.io/en/latest/going_further.html#localization-only):
TRAJECTORY_BUILDER.pure_localization_trimmer = {   max_submaps_to_keep = 3, }
I've also messed around with other parts of the config, but no matter what I do, Cartographer keeps mapping. I did notice that if the robot moves to a different submap, the modified submap will revert to it's original state but I want Cartographer to not add anything to the global map at all.


Update authentication policy tasks

Make tutorial for enable/disable mTLS clearer for different scopes of authentication policy and the role of destination rules.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: diemtvu
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: sdake
If they are not already assigned, you can assign the PR to them by writing /assign @sdake in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

content/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/assign @liminw

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: diemtvu, liminw
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: sdake
If they are not already assigned, you can assign the PR to them by writing /assign @sdake in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

content/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

New changes are detected. LGTM label has been removed.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: diemtvu, liminw
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: linsun
If they are not already assigned, you can assign the PR to them by writing /assign @linsun in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/assign @frankbu

Also update End-user authn tutorial for ingress setup (issue https://github.com/istio/istio.github.io/issues/1896)

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: diemtvu, liminw
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: frankbu
If they are not already assigned, you can assign the PR to them by writing /assign @frankbu in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/retest


bk gets stuck

I tried downloading and running the cli for a local build, but it just sat there doing nothing for 10 minutes until I figured it got stuck somehow and hit ^C:
karl@chrx:~/work/goprofile_test$ bk local run
>>> Starting local agent 🤖
>>> Starting build 👟
>>> Executing initial command: buildkite-agent pipeline upload
^C
>>> Gracefully shutting down...
🚨 Initial command failed: context canceled

When I ran it with debug, it complained about permission problems, and then got stuck again:
karl@chrx:~/work/goprofile_test$ bk local run --debug
2019/12/03 13:25:51 [keyring] Considering backends: [pass file]
2019/12/03 13:25:51 [keyring] Failed backend pass: The pass program is not available
2019/12/03 13:25:51 Serving API on http://<IP_ADDRESS>:35269
2019/12/03 13:25:51 Creating bootrap script at /tmp/bootstrap-458222797
>>> Starting local agent 🤖
>>> Starting build 👟
>>> Executing initial command: buildkite-agent pipeline upload
open /etc/buildkite-agent/buildkite-agent.cfg: permission denied^C
>>> Gracefully shutting down...

🚨 Initial command failed: context canceled

How do I fix this?

Running as root worked for me.

The local run isn't in 3.0.0 and we don't currently have a view to add it in; as has been seen, keeping it in line with the ongoing changes in the agent can mean it falls out of sync. We'll be looking at alternative approaches to using local run.


A typo in ec2.md

Describe the bug
A typo in ec2.md,
https://github.com/NVIDIA/spark-rapids-examples/blob/branch-21.12/docs/get-started/xgboost-examples/csp/aws/ec2.md#step-41-edit-spark-defualtconf
defualtconf => defaultconf

closed since #73 merged


Validate duplicate roles via a validating webhook

While backfilling tests for roles we noticed that role duplication validation is implicitly implemented in the repository via generating stable role binding names and relying on not being able to create role bindings with duplicate names.
While this works for API users, kubectl users are still allowed to create identical role bindings (with different name though) which is not great. In order to address this we should introduce a validating webhook to ensure that a rolebinding for user X to role Y does not exists already.
Dev notes
Once the validation has been implemented via a webhook, get rid of the duplicate roles integration tests from roles_test.go

I have some thoughts and questions regarding this

feel like this needs to be a Feature or a Bug as it affects kubectl users.
we would be watching a core k8s resource across not just cf related but all namespaces. What sort of performance implications will it have on the controllers?
additional logic in the validating webhook to check duplicates only if the role binding created/modified/deleted is in a cf namespaces.
need to handle role bindings with multiple subjects/users
in terms of priority, as we usually target cf cli/api user functionality, maybe its ok to push this lower in the list of things to do.
what could go wrong if we do not implement this?


Update hu.js

Some language improvements...

Accepted. Will be deployed soon (it is late in the day here, don't want to cause problems overnight :) )

Thanks!
;)

Accepted. Will be deployed soon (it is late in the day here, don't want to cause problems overnight :) )
—
Reply to this email directly or view it on GitHub https://github.com/darkskyapp/forecast-io-translations/pull/57#issuecomment-163019579.


Working with other emoji fonts?

I am trying to apply Noto Color Emoji font. But getting NotImplementedException by component from Typography.OpenFont. have you ever tried noto font series? I am not familiar with font (i.e opentype tables, etc.)
I have tried NotoColorEmoji-Regular.ttf and NotoColorEmoji_WindowsCompatible.ttf.
Sorry for messy submit. I accidently pressed enter :(

I'm currently facing a similar issue with Noto Color Emoji. The regular TextBlock doesn't show the emoji at all (not an issue for this project) and the TextBlock from emoji.wpf just renders the Segoe UI emoji and ignores the Noto font.

I'm currently facing a similar issue with Noto Color Emoji. The regular TextBlock doesn't show the emoji at all (not an issue for this project) and the TextBlock from emoji.wpf just renders the Segoe UI emoji and ignores the Noto font.

I'm done writing code rendering text into bitmap via DirectWrite with SharpDX. It is a waste of memory and can't utilize hardware acceleration but it works with any fonts provided.


Things I'd like to work on

Hi guys. I'd like to explain some context for my interest in Viskell, areas I'm interested in working on, and get feedback on the them (to avoid working on stuff that would then never get merged).
After some discussion here I would close this issue and open some new ones on specific areas, each with limited scope, for more detailed discussion on specific subjects. Then you could assign to me.
Background
I want to explore writing/describing programs in a way that is not procedural, where the dependencies between statements and blocks of statements are explicit (via "data flows" between them) and that alone determines execution order. These would be descriptions of what has to be done, not how to do it, not explicitly what steps in what order and in how many threads etc.
Then I want to explore how to execute them in a highly parallel fashion on single core, multicore, and even across machines.
The language of the individual statements/blocks is not so important to me. A functional language where each block/function/lambda has no side effects fits well.
A strongly typed language that compiles also fits well. Ideally I'd like something that executes faster than Haskell seems to do....but that may come later.
The lazy execution "only execute a block when it's inputs are satisfied and it's output is requested" model of Haskell also fits.
Viskell
I started implementing something myself. I investigated other commercial apps that help you describe nested data flows (most are left overs from '90s focus on Structural Analysis and Design) and then export them into XML. I was reading the XML then in my app. But the visual tools came up short, were parts of massive toolsets for other purposes and I had no way to modify them.
Creating the whole visual tool to describe such apps was a big job, and not core to what I wanted to do.

I would like to be able to run them outside of viskell once described and saved, even if interpreted or slow, or single core initially.
Work required on Viskell
These are the areas that I think are needed and I propose to work on in Viskell:
IO and Main
To describe even the simplest app ("Hello World!") that can be executed outside of Viskell, you need to have it wrapped in a main, and so have some way of describing IOActions.
I'm still reading up on Haskell's IOActions, but would like to implement at least something basic (print to stdout, read from standard input or a named file) to enable some real-world apps with visible "side effects" for the user.
In the Viskell app it would involve adding some additional window/pane that shows stdout and/or stderr, and maybe one for getting input from the user when required. Also requires adding some IOActions and the ability to include them in the toolbar .
## Generate Source
Would be good to extend the generated source so it generated a main and hence an executable app. Alternatively, a separate tool/command that took the saved .viskell file and generated source or ran it.
Nested code
To be able to describe bigger apps, share code between apps, make design of large apps more manageable, and even have shared libraries of code - Viskell needs a way to incorporate nested code described in Viskell (and maybe Haskell, to leverage existing code libraries?).
The idea is that the file description would define the inputs/outputs at the outer boundary and that is what is available to the apps incorporating them to be connected to in Viskell.
Thus you could describe a mega-block and save it, without any IOActions or main - and it could be used nested in another mega-block or app that has IOActions and Main.

There are quite a few interesting decisions to be made (or avoided) here:
First off, do we want to be able to call all of Haskell? At the moment, we restrict ourselves to a subset using the catalog - a growing subset, but still a subset. If we want to be able to call all of Haskell, I guess we will likely have to be able to understand all of Haskell (more interesting types and so on).
Second, live evaluation. Live evaluation makes a lot sense for a pure expression language with only constants, like in the earliest versions of Viskell. It makes less sense when that language has lambda abstraction, as you can't say what the value of some x inside a lambda will be; Viskell doesn't allow observe blocks inside lambdas for that reason. When you add side effects, stuff gets even more complicated. (You don't want the 'format the hard drive' program you're writing to run every time you make a change.) Of course, in practice it could be that large parts of people's programs could be seen as pure expressions with only constants (i.e. by giving constant test data as an input), but this might be a bit of a puzzle.
What we could perhaps do as a (small) first step towards what you're describing is dump the Haskell code to a file when the user hits save, adding exports for all named lambdas. In real-world projects, the code with side effects (your main function and so on) would then be ordinary Haskell, and the pure, side-effect free code would be partially written in Viskell.
module Main where

import MyPureViskellModule
import FrobnicateLibrary (frob)

main :: IO ()
main = interact $ myViskellLambda frob

So Viskell would save a .viskell file and generate a .hs file with pure functions. ghc --make would ignore the first file, pick up the second, and build the final executable/library from that.
It wouldn't help the use-Haskell-code in Viskell case, or the use-Viskell-code-in-Viskell case, but it would make it possible to use Viskell code in a Haskell project, which will then run outside of Viskell (multi-threaded and so on).

So, focusing on one of those points you comment on....
Maybe something useful, and pragmatic but that could be worked on sooner rather than later without need for a lot of discussion and huge changes could be:
Saving of source to a file, in a way that it can be imported in a main and used to build a complete app.
Maybe also generating source for a Main capable of using it?
Then you could just "runhaskell main.hs"

Did a bit of reading today and discovered that there is A LOT of commonality with what I wanted to do, with TensorFlow from Google (https://www.tensorflow.org/).... although I was thinking more general and not so focused on Machine Learning.
http://download.tensorflow.org/paper/whitepaper2015.pdf
Note that the "function library" idea (having blocks of code be sharable across multiple "apps") is still on their future list.

All your plans are compatible with the longterm goals of Viskell, thus no need to worry about whether they will mergable. However they are focussed on the practical application of Viskell, and I personally think Viskell has still too many big gaps in the basic functionality for that. I do not want to discourage you from working on longer term features, but the question is what things could be done first to make Viskell more usable.
In my opinion the big shorter term open issues are:

adding more constructs to make Viskell a 'complete' programming language
making the user interface take a lot less actions to construct a program
scaling to larger programs in a practical way (beyond a single screen full)
increasing the density of the visualisation while keeping it readable
exploiting more feedback and interactivity using live compilation and execution

For IO code the idea is to use the Arrow abstraction from Haskell so that IO actions will be just a variant of functions blocks. Those IO block need to be in a special area and/or have some decorations to distinguish them from normal functions. Also the ordering needs to be indicated somehow. Supporting IO will very useful, but it will be quite hard to implement as it requires changes in basicly all aspects of Viskell.
I think live evaluation should work everywhere as a way of debugging, but it will be not ease to implement and may involve having two code generators (one for final program, and one for live stuff). For lambda's the issue is multiple runs of the same code with less defined orderings due to lazy evaluation, thus something is needed to display multiple values and/or give user control over which are relevant. Live execution of most IO actions could implemented using an alternative interpretation on a virtualized world, and a button somwhere to trigger the real world execution.
As for nested code we now only have lambda blocks that can be given a name and be reused within the same program. For larger programs we need more than that, but I have no concrete ideas yet of how that should work and look. The open questions to focus on are: what actions on nested code can a user perform, how to generate code for these constructs (library support?), and how to visualize it.
I have seen TensorFlow and it looks nice and polished, though the layouting of nodes seems a bit too automated. Instead of making a visual IDE for every specialized domain, we should improve on visual programming languages so all these things could be just another library.

Thanks for the thought put into your reply.
Here I comment on a few points, re-ordering them to comment on some together.

adding more constructs to make Viskell a 'complete' programming language

That seems like one better suited to the core team that developed it, which knows it better and probably Haskell better too.
From what you have said before, sounds like disconnecting viskell from haskell, or the UI from the graph model and it's evaluation is out of scope (I have thoughts on that though if you're interested).

For IO code the idea is to use the Arrow abstraction from Haskell so that IO actions will be just a variant of functions blocks. Those IO block need to be in a special area and/or have some decorations to distinguish them from normal functions. Also the ordering needs to be indicated somehow. Supporting IO will very useful, but it will be quite hard to implement as it requires changes in basicly all aspects of Viskell.

This seems like a fundamental one, in order to be able to describe useful programs with side effects.
I still need to understand how Haskell evaluates IO actions to know how difficult this is, the use of 'do' to group IO Actions, how they are evaluated when they fall in Main, etc etc....

exploiting more feedback and interactivity using live compilation and execution

Related to IO (above), Input and output can't just be done infinitely in a loop (e.g. reading input from a file and processing it, or writing to output files. STD input can block also - although that might not be an issue). For the "auto-eval" thing, I would just have a small UI control consisting of an auto/manual switch, and a "play button" that runs it all once when in manual mode and that is disabled in auto-mode. Maybe the addition of certain types of operators or blocks would move it into manual mode?

scaling to larger programs in a practical way (beyond a single screen full)

As for nested code we now only have lambda blocks that can be given a name and be reused within the same program. For larger programs we need more than that, but I have no concrete ideas yet of how that should work and look.

Embedded / Referenced viskells
My proposal here is double:

exploding nested blocks within the same Viskell. It shows as one block on the UI. You can expand it and zoom in (like TensorFlow) on the same same window. Or you can double-click it and open it for editing in a separate window. The serialization format would keep them in a separate segment of the same file and refer to it - so you can do kind of "breadth first" deserialization....

referenced viskell files. A block can be a reference to a separate Viskell file, developed by someone else and saved separately. On loading, we will have to check for consistency of the connections into and out of the block. This should enable code reuse and arbitrarily complex programs, where you "just trust" an imported block when you are focussing on your. If you double-click on them, they are opened in another window as a separate viskell. In Mac, it would become a "multi-document application" like a text editor.
Possibly, when saving such a block you mark the anchors/wires that will be available from outside it. i.e. the data that flows into and out of it. This is where I started originally, using the "Data flow diagram" metaphor, that checks consistency of data flowing into and out of each "bubble" and where bubbles can be exploded infinitely.

Related to the use of a referenced viskell file: We should try and use the same or similar mechanism for being able to use existing code libraries (Haskell in this case) and to generate code that imports them and calls them.

Separation
A separate one that excites me is the ability to have a tool that reads viskell files and evaluates them (or compiles them). This would read recursively any other embedded or references viskell files or haskell libraries to create the entire graph and compile/execute it.
In my opinion the best way to go would be to have a libraries that:

can read/write the graph model
evaluate a model live via gchi
take a graph model and convert to haskell.
These library sub-projects would be used by the UI "design" app interactively to do what it does not....and non-interactively by a "runner".

They could be used by a web-app implementation without the existing UI (something I have heard you refer to).
This requires separating view and model logic, where each "Block" type in the current code would just be javafx panes, that are built using Model objects for each block/wire. There would need to be an overall graph model that stores the blocks and (logical) wire connections, not have them held just by the javafx Main overlay.
Model objects for Block and Wire would be able to serialize and deserialize themselves like now, but without UI code needed to do so.
I think the creation of model objects for each block type and their use by the existing View code could be done gradually, without breaking the current app. Once all done, then a break would be needed where they were split into a library and the app that uses it.
I could try a WIP PR to show an example of that for a few Block types if you wanted....
Continuing the discussion
Lots of good discussion. Maybe we need to separate into distinct GH issues for each area to avoid mixing them all up?

adding more constructs to make Viskell a 'complete' programming language

That seems like one better suited to the core team that developed it, which knows it better and probably Haskell better too.
From what you have said before, sounds like disconnecting viskell from haskell, or the UI from the graph model and it's evaluation is out of scope (I have thoughts on that though if you're interested).

Most of the constructs viskell are not very haskell specific, any modern function programming language with an expressive enough type system would suffice. I see using haskell just as a convenient starting point, so we don't also have to invent complete new programming language at the same time. We try to avoid introducing haskell syntax and other complexities into viskell.
In the long term viskell could move towards being a superset of multiple languages, where different libraries and code generator can give alternative concrete interpretations of common set of visual constructs. We did some experiments with generating hardware descriptions already.
Moving the concrete ideas to seperate issue is a good idea.
I hope @wandernauta can give you some feedback on the scaling/nesting/loading aspects, as he has spent more time on thinking about these issues.


Expensify Card - Blue outline does not wrap the user row properly when navigating via Tab

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 9.0.4-0
Reproducible in staging?: Y
Reproducible in production?: N
If this was caught during regression testing, add the test name, ID and link from TestRail: N/A
Issue reported by: Applaus - Internal Team
Action Performed:

Go to staging.new.expensify.com
Go to workspace settings > More features
Enable Expensify Card
Navigate through user list via Tab key

Expected Result:
The blue outline will wrap the user row properly
Actual Result:
The blue outline does not wrap the user row properly
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[ ] Android: mWeb Chrome
[ ] iOS: Native
[ ] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[x] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence

https://github.com/Expensify/App/assets/78819774/0078e80b-1ea8-4626-92f6-1ca40a8d2c50
View all open jobs on GitHub

@puneetlath  FYI I haven't added the External label as I wasn't 100% sure about this issue. Please take a look and add the label if you agree it's a bug and can be handled by external contributors

I am demoting this one since it doesn't impact the user and tab navigation is not a mainline flow.

roposal
Please re-state the problem that we are trying to solve in this issue.
Expensify Card - Blue outline does not wrap the user row properly when navigating via Tab
What is the root cause of that problem?
We have incorrectly configured the styles on WorkspaceCardListRow:
https://github.com/Expensify/App/blob/77d380ce7d1fcaad2045a3c590dc16a1363d4cd1/src/pages/workspace/expensifyCard/WorkspaceCardListRow.tsx#L41-L43
We need to update these, so that blue border is applied correctly.
What changes do you think we should make in order to solve the problem?
We should have styles.selectionListPressableItemWrapper as the wrapping styles on the Pressable here
We should update the view styles here
<View style={[styles.flexRow, styles.flex1, styles.justifyContentBetween, styles.userSelectNone, styles.alignItemsCenter, style]}>

We should update the contentContainerStyle as styles.mb3 for the OfflineWithFeedback here
We will do other small adjustments to clean up the styles.
Branch here
Result
https://github.com/Expensify/App/assets/64629613/190aa1aa-a6d6-4818-b5a4-b73ca8e85480

Taking this on @koko57 @narefyev91 @allgandalf @DylanDylann
Can you please handle this polish. thanks

@allgandalf @narefyev91 It seems this issue relates to your PR

Please assign to me - will pick it up


fix: process.cwd error

在调用build/util/loadEnv()函数时会报process.cwd() is not a function
#44

please rebase merge main


ValueError: expected sequence of length 3338 at dim 1 (got 3608)

System Info / 系統信息
I am trying to finetune the GLM-9B-Chat model with axolotl. I am getting the following error:
Traceback (most recent call last):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 59, in <module>
    fire.Fire(do_cli)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
    return do_train(parsed_cfg, parsed_cli_args)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 55, in do_train
    return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/train.py", line 163, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/trainer.py", line 1859, in train
    return inner_training_loop(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/trainer.py", line 2165, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/accelerate/data_loader.py", line 452, in __iter__
    current_batch = next(dataloader_iter)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/monkeypatch/data/batch_dataset_fetcher.py", line 32, in fetch
    return self.collate_fn(data)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/utils/collators.py", line 106, in __call__
    features = self.tokenizer.pad(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3355, in pad
    return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 224, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 775, in convert_to_tensors
    raise ValueError(
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`position_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
Traceback (most recent call last):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 759, in convert_to_tensors
    tensor = as_tensor(value)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 721, in as_tensor
    return torch.tensor(value)
ValueError: expected sequence of length 3509 at dim 1 (got 3573)

pytorch: 2.0.1
Cuda: 11.8
transformers: 4.40.0.dev0

Who can help? / 谁可以帮助到您？
No response
Information / 问题信息

[ ] The official example scripts / 官方的示例脚本
[X] My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程
Setup the axolotl framework from it's original github repository, and use the following config file.
base_model: THUDM/glm-4-9b-chat
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

trust_remote_code: true

load_in_8bit: true
load_in_4bit: false
strict: false

datasets:
    - path: alpaca.json
      ds_type: json
      type: alpaca
dataset_prepared_path: glm/
val_set_size: 0
output_dir: ./outputs/glm-lora

sequence_len: 2048  # supports up to 8192
sample_packing: false
pad_to_sequence_len: true

adapter: lora
lora_model_dir:
lora_r: 62
lora_alpha: 32
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project: ArabicGLM-8B
wandb_entity:
wandb_watch:
wandb_name: ArabicGLM-8B
wandb_log_model:

gradient_accumulation_steps: 8
micro_batch_size: 4
num_epochs: 1
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention:

warmup_steps: 95
evals_per_epoch: 4
eval_table_size:
eval_max_new_tokens: 128
save_total_limit: 5
save_steps: 200

debug:
deepspeed: deepspeed_configs/zero2.json
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
  pad_token: "<|endoftext|>"
  eos_token: "<|endoftext|>"

Expected behavior / 期待表现
It should start finetuning the model but it's throwing the error:
Traceback (most recent call last):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 59, in <module>
    fire.Fire(do_cli)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
Traceback (most recent call last):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 759, in convert_to_tensors
    tensor = as_tensor(value)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 721, in as_tensor
    return torch.tensor(value)
ValueError: expected sequence of length 3781 at dim 1 (got 3446)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return do_train(parsed_cfg, parsed_cli_args)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 55, in do_train
    return _run_code(code, main_globals, None,
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 59, in <module>
    return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/train.py", line 163, in train
    fire.Fire(do_cli)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 35, in do_cli
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/trainer.py", line 1859, in train
    return inner_training_loop(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/trainer.py", line 2165, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/accelerate/data_loader.py", line 452, in __iter__
    current_batch = next(dataloader_iter)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/monkeypatch/data/batch_dataset_fetcher.py", line 32, in fetch
    return do_train(parsed_cfg, parsed_cli_args)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/cli/train.py", line 55, in do_train
    return self.collate_fn(data)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/utils/collators.py", line 106, in __call__
    return train(cfg=cfg, cli_args=cli_args, dataset_meta=dataset_meta)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/train.py", line 163, in train
    features = self.tokenizer.pad(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3355, in pad
    return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 224, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 775, in convert_to_tensors
    raise ValueError(
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`position_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/trainer.py", line 1859, in train
    return inner_training_loop(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/trainer.py", line 2165, in _inner_training_loop
    for step, inputs in enumerate(epoch_iterator):
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/accelerate/data_loader.py", line 452, in __iter__
    current_batch = next(dataloader_iter)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/monkeypatch/data/batch_dataset_fetcher.py", line 32, in fetch
    return self.collate_fn(data)
  File "/mnt/beegfs/fahad.khan/axolotl/src/axolotl/utils/collators.py", line 106, in __call__
    features = self.tokenizer.pad(
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 3355, in pad
    return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 224, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "/home/ashmal.vayani/anaconda3/envs/axolotl/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 775, in convert_to_tensors
    raise ValueError(
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`position_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (position_ids in this case) have excessive nesting (inputs type list where type int is expected).
看样子是你做数据集处理的时候长度没有对齐，是否可以检查一下这个部分呢，因为这个框架我没有尝试过

I am not sure why this error arises as for the other models, this framework adjusts the length itself for the other models.
I have also tried your finetuning script but it's behaving a bit weirdly. It ran for 3000 steps very fast but then it was stuck in dumping something in the tmp.
Also, can you share a data sample that works best with your script? I see that the data sample only supports multi-turn conversations.
Why are the max tokens and context length set for the glm model limited to 512 tokens?

A single turn is also a special kind of multi-turn interaction, but the turn length is just 1. Additionally, the 512 tokens you mentioned can be changed to an output of 4096 tokens. what kind of dataset do you want to finetune. Is it not a conversation format?

So I converted the single turn to the mentioned format in finetune_demo/readme.md file. I fine-tuned it on AceGPT Dataset for Arabic. After qlora, the performance decreased than the base accuracy. Although, the similar dataset has resulted in improved accuracy with other similar-sized models. By increasing the 512 tokens to higher, it goes OOM very quickly.
[
  {
    "messages": [
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      }
    ]
  },
{
    "messages": [
      {
        "role": "user",
        "content": "<user prompt text>"
      },
      {
        "role": "assistant",
        "content": "<assistant response text>"
      }
    ]
  },
 ...
]

By examining your data structure, it seems that there is no problem, so it seems that I cannot provide you with any further assistance. Additionally, if the length of your dataset does not reach 512, you can use a shorter length for fine-tuning.


Suggestion for improving encryption key text

Expected behavior
"end-to-end key" is a very unusual wording for encryption. Unless space is really restricted, this could read "Configure end-to-end encryption key"
Actual behavior
String account_settings_crypto_key reads "Configure end-to-end key"

It's "configure end-to-end key" now.


Add an action to open folds

I'm trying to setup a list mappings to be able to open a folded list with 'l' instead of enter. However, I did not see any action that would allow me to open folds. I saw a open action but this does not seem to do what I'd like to do and it errors out anyway with:
E5108: Error executing lua: ...k/myNeovimPackages/start/glance-nvim/lua/glance/init.lua:229: method: expected 'definitions'|'type_definitions'|'references'|'implementations', got nil
stack traceback:
        [C]: in function 'error'
        vim/shared.lua: in function 'validate'
        ...k/myNeovimPackages/start/glance-nvim/lua/glance/init.lua:229: in function <...k/myNeovimPackages/start/glance-nvim/lua/glance/init.lua:228>

Is there currently a way to achieve what I'd like to do?

Hi,
For that you should use actions.jump action, it toggles folds when the current selected item in the list is a fold.
I probably should specify that in the docs. Thanks.