test pull request

adding functions to track orientations issue #167

Hi @tfamprikis thanks for this! This looks like it will be quite a useful starting point. Do you plan on adding something more? Otherwise we will start integrating this into gemdat.

hey, this was just a test to see how this works.
i still have a couple of issues that havent been able to fix. mainly

I cant seem to be able to symmetrize my directions (created by gr.bond_direct() ). this is what I'm trying to do in lines 28-59 of test4.py. It works in matlab but not in python...
I've trouble with normalizing the probability plot (in polar coordinates) by the areas of the sphere, it becomes problematic near the poles, creating artifacts.

otherwise some other functions from test4.py can also be moved to gemdat_rotation (autocorr(), autocorr_par(), a function to plot distribution of bond lenghts)
I can add the data (.xml) i've been using to test to test_data and some figures of what the symmetrization looks like when it works.
all that said, if you think you can still work with what I have please go ahead...

Okay, no worries! The test at leas worked 😅
There is a bunch of othes stuff I still want to work on as well. I'll let you know when we get back to this.

@tfamprikis Can you share the data that you used to run test4.py?
Also, have you made any further progress about the rotations that is not already here?

This pull request should fulfil the following objectives:

[x] compute the orientation vectors and their trajectory given the input expected structure
[ ] apply the correct symmetry group to enhance the trajectory
[ ] produce the rotolinear plot
[ ] measure and plot the autocorrelation

The analysis of rotational transitions will be the object to the next pull request

You can look at this notebook to get an overview of this addition.

We could use:
from pymatgen.symmetry.groups import PointGroup
g = symmetry.groups.PointGroup('m-3m')  # Oh
len(g.symmetry_ops)
# 48
g.symmetry_ops[0]
# SymmOp(affine_matrix=array([[ 0.,  0., -1.,  0.],
#       [ 0., -1.,  0.,  0.],
#       [-1.,  0.,  0.,  0.],
#       [ 0.,  0.,  0.,  1.]]))

I have cleaned the api, so now the basic usage looks like:
bd = Orientations(trajectory, central_atoms='S', satellite_atoms='O', n_expected_neigh=8, normalize_trajectory=False)

plots.rectilinear_plot(data=bd, symmetrize=False)

bd.set_symmetry_operations(sym_group='m-3m')

plots.rectilinear_plot(data=bd, symmetrize=True)

which is now using pymatgen.PointGroup to get the symmetry, so I have removed Oh_point_group.
I have also refactored and removed some superfluous methods as suggested.

Hi guys, I've been playing with this a while. some implementations don't really make sense to me rn. should I comment here or wait for this to be merged and make issues?

@tfamprikis I think this pull request will be closed relatively soon (since it is also quite long), so it would be nice if you could open new issues to address all the problems that you see.


Elasticsearch version

I asked this in the Google Group, but is there a reason why Blueflood is pinned to Elasticsearch 1.7, and are there any plans to upgrade soon? It's 3 major versions behind at this point.

We have plans to upgrade ES. We are currently working on upgrading to 2.4.1 and then to 5.0. ES recommends to do it in multiple steps.
https://github.com/rackerlabs/blueflood/pull/808
I dont have any timelines yet but somewhere down the line.

Hey there @DanielNi.  I almost have Blueflood working against ES 5.4.2.  The main blocker is that the tests for blueflood-elasticsearch need to be entirely rewritten.  Just seeing if you're still interested in this.
Anecdotally, I also have things running against Cassandra 3.11.1.

Thanks for the update @stackedsax. I don't think this is high priority for us anymore.


User Stories for Game API Endpoint Development for Retrieving and Posting Game Data

Goal: To create user stories for a GET endpoint at /api/games to retrieve a list of existing games, providing essential information such as game titles, genres, and release dates. This will empower users and external systems to easily access and display game data. To create user stories for a POST endpoint at /api/games to enable the addition of new games to the system. This feature will facilitate the submission of game details, including titles, genres, and release dates, contributing to the expansion and enrichment of the game database.
These user stories drive the implementation of a versatile API endpoint, enhancing the overall functionality of the chigame platform by enabling seamless access to existing game data and facilitating the addition of new games through a standardized and efficient process.

POST Games Endpoint:
-Admin user Michael wants to create a game with a minimum of 1 player and maximum of 4. The game is titled Upturn and has a very generic description. He uploads an image that will be displayed whenever users view the game, and states the rules of the game. The suggested age is 10+ and will be inputted as 10. The expected playtime will be set to 45 minutes with no minimum or maximum specified. The complexity of the game will be set to 2, and the category is listed as "Other", with the uplift and upturn mechanics being highlighted. Upon posting, the game will receive its Board Game Geek ID which is autoincremented by django.
-Users that opt to will automatically be notified of this incredible new game through an automated GET API. They will be able to see the cover image of this game, its rules, description, complexity, genre, mechanics, minimum and maximum players, the age restriction, and estimated playtime.

POST Games Endpoint:
-Admin user Michael wants to create a game with a minimum of 1 player and maximum of 4. The game is titled Upturn and has a very generic description. He uploads an image that will be displayed whenever users view the game, and states the rules of the game. The suggested age is 10+ and will be inputted as 10. The expected playtime will be set to 45 minutes with no minimum or maximum specified. The complexity of the game will be set to 2, and the category is listed as "Other", with the uplift and upturn mechanics being highlighted. Upon posting, the game will receive its Board Game Geek ID which is autoincremented by django.
-Users that opt to will automatically be notified of this incredible new game through an automated GET API. They will be able to see the cover image of this game, its rules, description, complexity, genre, mechanics, minimum and maximum players, the age restriction, and estimated playtime.
GET Games Endpoint:
As a game enthusiast, Shannon want to retrieve detailed information about the recently posted game "Upturn" through a GET API endpoint at /api/games, so she can stay informed about new and exciting additions to the chigame platform.

The GET API endpoint at /api/games successfully retrieves information about the game "Upturn."
She can view the cover image, rules, description, complexity, genre, mechanics, minimum and maximum players, age restriction, and estimated playtime for the game.
The retrieved data includes the auto-incremented Board Game Geek ID assigned to the game by Django.


Roles authentication not working

I have my site set up to only allow users with the "admin" role to be able to use the CMS. However, anyone seems to be able to sign in regardless of role. My config looks like this:
backend:
  name: git-gateway
  branch: gatsby
  accept_roles:
    - admin

I'm using Netlify Identity. I've signed up for new accounts with no roles whatsoever, but they can still sign in.

Nevermind. I didn't realize you also have to set the role in the gateway options (in Netlify, not the config) as well as the user options.


Error handling

Tasks

[x] Add axios and related error handling to remaining API calls
[x] Implement error handling
[x] Update Error component to match mocks

Complete


WidgetCommon not defined?

Hello, I was poking around the examples and noticed that the tictactoe example did not seem to work on my machine and provided the following error.
ERROR: LoadError: UndefVarError: WidgetCommon not defined
Stacktrace:
 [1] top-level scope at /home/leios/.julia/packages/Parameters/CVyBv/src/Parameters.jl:572
 [2] top-level scope at REPL[1]:1
in expression starting at /home/leios/projects/TextUserInterfaces.jl/examples/tictactoe.jl:28

I found WidgetCommon being exported in src/types/widgets.jl, but the definition does not seem to be in that file.

How did you add the package? The package is under development, so I would suggest adding the master branch. Something like ] add TextUserInterfaces#master
See related issue: https://github.com/ronisbr/TextUserInterfaces.jl/issues/6

Yes! @affans is completely right! However, I am doing a major change to improve the loading time. I might have forgotten to update Tic Tac Toe. I will try to push the changes today or tomorrow, and then everything will be working again. Sorry about that :)

I did
Pkg> add TextUserInterfaces#master

So it should be the master branch

So, I really forgot to update this example to use the new interface. I will try to fix this until tomorrow.

Cool, thanks! No real rush, just poking around a bit. I should hopefully be using the package for some somewhat complicated stuff in the next few weeks, but we'll see.
Thanks for the active development!

Yes! It is in active development. It just turns out to be extremely complicated to create a API for TUIs. If you want to use NCURSES directly, then it is more or less good to go.

Right now, ncurses is fine, what else would you be considering adding? Sorry, I don't see a project open or anything

I want to create an Julia-like API. So, instead of drawing things on screen (lines, text, etc.), I would like to provide widgets, focus manager, anchoring system to place the widgets, etc. More or less what we have when developing a Qt software (but waaaaaaaaay more simplified :D)

Done! Should be working now :)

Just one information, I have update many things internally. Now we have some useful signals and a new widget called Canvas. Hence, the custom widget in tic tac toe example is no longer needed. You will see that the code is much cleaner now.


Adding functionality to read additional pages of device info

Description
Adding functionality to read additional pages of device info
Fields added

NfcRestricted

Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How has this been tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Test configuration:

Firmware version:
Yubikey model:

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have run dotnet format to format my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

fyi @equijano21


Slow training

I'm trying to train BART from HuggingFace library.
But the training is extremely slow, on 4 GPU. Logs show this :

[INFO 2020-06-08 05:26:19] 0/2650, SamplesPerSec=13.19749106827976
[INFO 2020-06-08 05:26:34] 0/2700, SamplesPerSec=13.198177991591365

It says 13 samples per sec, but if we look at the steps number and log time, we can see it takes 15 sec to do 50 samples =>  3.33 samples per sec.
Why is the training so slow and why the logs are showing contradicted information ?

Can you enable wall_clock_breakdown in your deepspeed json config? https://www.deepspeed.ai/docs/config-json/#logging
This breakdown should give us some additional information on where the time is being spent.

Thanks for the lightning-fast answer !
Here is some logs with the wall_clock_breakdown enabled :

[INFO 2020-06-08 06:02:19] 0/450, SamplesPerSec=7.458824487454903
time (ms) | forward_microstep: 193.65 | backward_microstep: 329.25 | backward_inner_microstep: 329.14 | backward_allreduce_microstep: 0.05 | step_microstep: 0.92
time (ms) | forward_microstep: 203.21 | backward_microstep: 323.86 | backward_inner_microstep: 323.81 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 195.00 | backward_microstep: 325.61 | backward_inner_microstep: 325.56 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 186.87 | backward_microstep: 330.72 | backward_inner_microstep: 330.64 | backward_allreduce_microstep: 0.04 | step_microstep: 0.05
time (ms) | forward_microstep: 183.16 | backward_microstep: 334.45 | backward_inner_microstep: 334.34 | backward_allreduce_microstep: 0.05 | step_microstep: 0.08
time (ms) | forward_microstep: 186.46 | backward_microstep: 338.38 | backward_inner_microstep: 338.33 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 199.31 | backward_microstep: 335.63 | backward_inner_microstep: 335.56 | backward_allreduce_microstep: 0.03 | step_microstep: 0.05
time (ms) | forward_microstep: 192.47 | backward_microstep: 338.06 | backward_inner_microstep: 337.95 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 192.41 | backward_microstep: 333.76 | backward_inner_microstep: 333.70 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 189.00 | backward_microstep: 338.92 | backward_inner_microstep: 338.81 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 186.59 | backward_microstep: 341.10 | backward_inner_microstep: 341.04 | backward_allreduce_microstep: 0.03 | step_microstep: 0.03
time (ms) | forward_microstep: 193.71 | backward_microstep: 343.33 | backward_inner_microstep: 343.22 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 193.71 | backward_microstep: 341.28 | backward_inner_microstep: 341.21 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 201.06 | backward_microstep: 333.03 | backward_inner_microstep: 332.95 | backward_allreduce_microstep: 0.03 | step_microstep: 0.05
time (ms) | forward_microstep: 196.37 | backward_microstep: 850.88 | backward_inner_microstep: 326.98 | backward_allreduce_microstep: 523.86 | step_microstep: 193.84
time (ms) | forward: 3079.24 | backward: 5859.61 | backward_inner: 5334.38 | backward_allreduce: 524.09 | step: 195.15
time (ms) | forward_microstep: 193.11 | backward_microstep: 319.56 | backward_inner_microstep: 319.51 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 182.26 | backward_microstep: 338.71 | backward_inner_microstep: 338.64 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 190.97 | backward_microstep: 341.18 | backward_inner_microstep: 341.11 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 203.62 | backward_microstep: 332.63 | backward_inner_microstep: 332.58 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 195.21 | backward_microstep: 331.61 | backward_inner_microstep: 331.56 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 204.40 | backward_microstep: 325.38 | backward_inner_microstep: 325.25 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 188.99 | backward_microstep: 343.44 | backward_inner_microstep: 343.36 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 187.01 | backward_microstep: 328.58 | backward_inner_microstep: 328.49 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 189.68 | backward_microstep: 331.94 | backward_inner_microstep: 331.86 | backward_allreduce_microstep: 0.03 | step_microstep: 0.05
time (ms) | forward_microstep: 193.26 | backward_microstep: 332.35 | backward_inner_microstep: 332.30 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 192.74 | backward_microstep: 330.04 | backward_inner_microstep: 329.98 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 191.39 | backward_microstep: 334.16 | backward_inner_microstep: 334.04 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 185.21 | backward_microstep: 331.86 | backward_inner_microstep: 331.81 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 194.69 | backward_microstep: 357.35 | backward_inner_microstep: 357.30 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 199.06 | backward_microstep: 332.70 | backward_inner_microstep: 332.58 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 195.33 | backward_microstep: 585.38 | backward_inner_microstep: 331.97 | backward_allreduce_microstep: 253.37 | step_microstep: 205.53
time (ms) | forward: 3086.26 | backward: 5596.61 | backward_inner: 5342.04 | backward_allreduce: 253.53 | step: 205.91
time (ms) | forward_microstep: 204.31 | backward_microstep: 347.69 | backward_inner_microstep: 347.64 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 203.15 | backward_microstep: 353.87 | backward_inner_microstep: 353.74 | backward_allreduce_microstep: 0.06 | step_microstep: 0.08
time (ms) | forward_microstep: 202.74 | backward_microstep: 338.03 | backward_inner_microstep: 337.90 | backward_allreduce_microstep: 0.06 | step_microstep: 0.08
time (ms) | forward_microstep: 194.12 | backward_microstep: 334.57 | backward_inner_microstep: 334.45 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 199.06 | backward_microstep: 357.45 | backward_inner_microstep: 357.37 | backward_allreduce_microstep: 0.04 | step_microstep: 0.05
time (ms) | forward_microstep: 206.16 | backward_microstep: 353.00 | backward_inner_microstep: 352.91 | backward_allreduce_microstep: 0.04 | step_microstep: 0.06
time (ms) | forward_microstep: 201.97 | backward_microstep: 334.28 | backward_inner_microstep: 334.16 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 189.21 | backward_microstep: 336.03 | backward_inner_microstep: 335.97 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 195.16 | backward_microstep: 338.33 | backward_inner_microstep: 338.28 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 194.98 | backward_microstep: 337.91 | backward_inner_microstep: 337.86 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 194.96 | backward_microstep: 334.13 | backward_inner_microstep: 334.02 | backward_allreduce_microstep: 0.05 | step_microstep: 0.07
time (ms) | forward_microstep: 199.71 | backward_microstep: 325.73 | backward_inner_microstep: 325.64 | backward_allreduce_microstep: 0.05 | step_microstep: 0.05
time (ms) | forward_microstep: 191.91 | backward_microstep: 350.42 | backward_inner_microstep: 350.33 | backward_allreduce_microstep: 0.04 | step_microstep: 0.06
time (ms) | forward_microstep: 193.48 | backward_microstep: 329.82 | backward_inner_microstep: 329.76 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 190.36 | backward_microstep: 330.53 | backward_inner_microstep: 330.45 | backward_allreduce_microstep: 0.03 | step_microstep: 0.05
time (ms) | forward_microstep: 190.08 | backward_microstep: 442.58 | backward_inner_microstep: 334.73 | backward_allreduce_microstep: 107.81 | step_microstep: 192.72
time (ms) | forward: 3150.64 | backward: 5544.11 | backward_inner: 5434.84 | backward_allreduce: 108.06 | step: 193.21
time (ms) | forward_microstep: 200.72 | backward_microstep: 349.60 | backward_inner_microstep: 349.53 | backward_allreduce_microstep: 0.03 | step_microstep: 0.04
time (ms) | forward_microstep: 205.10 | backward_microstep: 351.63 | backward_inner_microstep: 351.58 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
time (ms) | forward_microstep: 199.97 | backward_microstep: 348.74 | backward_inner_microstep: 348.68 | backward_allreduce_microstep: 0.02 | step_microstep: 0.03
[INFO 2020-06-08 06:02:47] 0/500, SamplesPerSec=7.421826531600121

SamplesPerSec is now 7 because I used 2 GPU instead of 4

I'm not really sure how to interpret this result. Anything seems out-of-place ?

Hi @Colanim, it looks like you are using gradient accumulation from those logs. You can see that the allreduce and step are close to zero for more steps - they are amortized over time by accumulating local gradients for multiple steps before reducing them and then updating model steps.
You can see from this line that only about a small amount of time is spent in the communication step (allreduce) since multiple forward/backward passes have been performed:
time (ms) | forward: 3079.24 | backward: 5859.61 | backward_inner: 5334.38 | backward_allreduce: 524.09 | step: 195.15

To accelerate training, if memory permits you can try to use a larger microbatch size to increase the amount of work you perform in each forward/backward pass.

Thank you for your answer.
Unfortunately, my GPU have a small amount of memory : I can't put more sample in a batch.
I'll close if nothing seems out of place. Thanks again for your answer !


Documentation window option border not working

FAQ

[X] I have checked the FAQ and it didn't resolve my problem.

Announcement

[X] I have checked Breaking change announcement.

Minimal reproducible full config
I configured both the window completion and documentation to have a custom border that I've defined. The  completion border works great, but the documentation is  still a black box with no border. The style is below, and the cmp configuration (using lazy,nvim) is below that:
Style
local style = {}

style.border = { '╭', '─', '╮', '│', '╯', '─', '╰', '│' }

return style

Cmp configuration
return {
	'hrsh7th/nvim-cmp',
	version = false, -- last release is way too old
	event = 'InsertEnter',
	dependencies = {
		'hrsh7th/cmp-nvim-lsp',
		'hrsh7th/cmp-buffer',
		'hrsh7th/cmp-path',
		'saadparwaiz1/cmp_luasnip',
		'hrsh7th/cmp-calc',
		'hrsh7th/cmp-emoji',
		'f3fora/cmp-spell',
		'kdheepak/cmp-latex-symbols',
		'jmbuhr/cmp-pandoc-references',
	},
	opts = function(_, opts)
		local cmp = require('cmp')
		local lspkind = require('lspkind')
		opts.sources = cmp.config.sources(vim.list_extend(opts.sources, {
			{ name = 'otter' },
			{ name = 'pandoc_references' },
			{ name = 'latex-symbols' },
			{ name = 'nvim_lsp_signature_help' },
			{ name = 'spell' },
			{ name = 'treesitter', keyword_length = 5, max_item_count = 3 },
			{ name = 'calc' },
			{ name = 'emoji' },
		}))
		-- stylua: ignore
		opts.window = {
		    completion = {
		        border = require('config.style').border
		    },
			documentation = {
		        border = require('config.style').border
			}
		}
		opts.formatting = {
			format = lspkind.cmp_format(),
		}
	end,
}

Description
I configured both the window completion and documentation to have a custom border that I've defined. The  completion border works great, but the documentation is  still a black box with no border.
Steps to reproduce
I've included all the relevant details.
Expected behavior
I've included all the relevant details.
Actual behavior
I've included all the relevant details.
Additional context
No response

@Kindlewing , did you find a solution for this bug?


AttributeError: 'function' object has no attribute '__func__'

I'm noticing crash in latest git commit.
safe: 01248253a3e8aedba6d473469dc839cd368bfe3c
crash: f31a338cbbdcd76a5af35e400eb9e0e8cae36b72
Command: accelerate launch scripts/finetune.py examples/openllama-3b/config.yml (point to proper config depend on commit)
INFO:root:Starting trainer...
Traceback (most recent call last):
  File "/workspace/scripts/finetune.py", line 321, in <module>
    fire.Fire(train)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/scripts/finetune.py", line 308, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1643, in train
    return inner_training_loop(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1750, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1182, in prepare
    result = tuple(
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1183, in <genexpr>
    self._prepare_one(obj, first_pass=True, device_placement=d) for obj, d in zip(args, device_placement)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1022, in _prepare_one
    return self.prepare_model(obj, device_placement=device_placement)
  File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/accelerator.py", line 1308, in prepare_model
    model.forward = MethodType(torch.cuda.amp.autocast(dtype=torch.float16)(model.forward.__func__), model)
AttributeError: 'function' object has no attribute '__func__'

Sure @anshsarkar .
I was testing a config here https://github.com/OpenAccess-AI-Collective/axolotl/blob/2ba4ae8f461c0c491f9ca303c134f9ad6f725e8c/examples/openllama-3b/config.yml on a machine where accelerate config precision is set to bf16 or fp16. This would cause a mismatch. I simply just changed the config to use None instead and it worked.
Vice versa, you can change the config.yml to match your accelerate's config.
Are you using axolotl or just accelerate in general?

@NanoCode012 I am using accelerate in general
Thanks for the input.  I will try this and see.

You can check your code where you cast to a type. Make sure it matches your accelerate's config. @anshsarkar

from accelerate import Accelerator
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, prepare_model_for_int8_training
from accelerate.utils import DistributedType
model_id = "ausboss/llama-30b-supercot"
kwargs = DistributedType("NO")
accelerator = Accelerator(device_placement=False, mixed_precision= "fp16" , cpu=False)
model = LlamaForCausalLM.from_pretrained(model_id, device_map=device_map_lm, load_in_8bit=True, torch_dtype=torch.float16)
model = prepare_model_for_int8_training(model)
model = accelerator.prepare(model)
training_arguments = transformers.TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    max_steps=20,
    learning_rate=LEARNING_RATE,
    fp16=True,
    logging_steps=1,
    optim="adamw_torch",
    evaluation_strategy="steps",
    save_strategy="steps",
    eval_steps=1,
    save_steps=1,
    output_dir=OUTPUT_DIR,
    save_total_limit=3,
    load_best_model_at_end=True,
    report_to="tensorboard",
    ddp_find_unused_parameters=False,
#   deepspeed = deepspeed_config
)
data_collator = transformers.DataCollatorForSeq2Seq(
    tokenizer, pad_to_multiple_of=8, return_tensors="pt", padding=True
)
trainer = transformers.Trainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=val_data,
    args=training_arguments,
    data_collator=data_collator,

)
model.config.use_cache = False
old_state_dict = model.state_dict
model.state_dict = (
    lambda self, *_, **__: get_peft_model_state_dict(
        self, old_state_dict()
    )
).__get__(model, type(model))

model = torch.compile(model)

trainer.train()
model.save_pretrained(OUTPUT_DIR)

@NanoCode012 for the above i am getting the error you encountered. Haven't been able to solve it yet. Your input will be really helpful.
Also thanks in advance!

accelerator = Accelerator(device_placement=False, mixed_precision= "fp16" , cpu=False)

Train the above line to the proper precision that you set when you run accelerate config

Hi, I face same problem here, my code to create Accelerator is the following:
accelerator = Accelerator()
My yaml look like this:
hyperparameters:   dataloader_drop_last: True   evaluation_strategy: "epoch"   save_strategy: "epoch"   logging_strategy: "epoch"   num_train_epochs: 10   auto_find_batch_size: True   batch_size: 4   max_steps: 1000   eval_steps: 100   save_steps: 1000   logging_steps: 100   per_device_train_batch_size: 8   per_device_eval_batch_size: 8   learning_rate: 1e-5   lr_scheduler_type: "cosine"   warmup_steps: 2000   gradient_accumulation_steps: 1   gradient_checkpointing: True   sharded_ddp: False   fsdp: False   weight_decay: 0.0001   run_name: "CodeT5-seq2seq-fine-tuned"   ddp_find_unused_parameters: False   fp16: True   bf16: False   auto_find_batch: True   num_workers: 4   max_prediction_length: 512   beam_size: 5   max_grad_norm: 5.0   adam_epsilon : 1e-06
Even I remove the fp16, bf16 settings or set them to False, the error still exists.
What should I do to make it work?

@Luxios22 , did you run accelerate config to make sure it matches with this yaml?

@NanoCode012
I tried matching it with the yaml and setting to None as well. Still getting the same error

I'm not sure then, sorry. That was what fixed for me. Make sure to try reinstall latest version of the packages as well.

Hmmmm, sure. Thanks for the help!!

@anshsarkar @Luxios22 I managed to get my script working by downgrading to transformers==4.29.2, seems like there were some changes from v4.30.0 onwards that introduced this issue. I opened an issue on the HF/transformers repo if you want to track it https://github.com/huggingface/transformers/issues/24431

This issue is not completed and should not be closed until at the very least there is an informative error message given.

Hey @enn-nafnlaus , did you try downgrade following Steven's advice? That worked for me on newer machines. Could you list your steps to reproduce? axolotl or not etc?

Hey @enn-nafnlaus , did you try downgrade following Steven's advice? That worked for me on newer machines. Could you list your steps to reproduce? axolotl or not etc?

Just got to that advice (was working my way down this thread) and it worked (might want to have that be the error message  :)  )... well, to the degree that it eliminated the 8th consecutive error in trying to get it to run. But now off to the 9th, which is three separate consecutive errors (that happen regardless of whether my yaml has load_in_4bit enabled):
ValueError: Autograd4bitQuantLinear() does not have a parameter or a buffer named weight.
ValueError: Autograd4bitQuantLinear() does not have a parameter or a buffer named weight.
TypeError: LlamaForCausalLM.init() got an unexpected keyword argument 'load_in_4bit'
Surely unrelated to the func error though.

Yes @enn-nafnlaus . This is the caveat of this method. 4bit wasn't implemented in this version.. If you're not using it, you can simply comment out the lines. Otherwise, I'm not sure how else.. It seems to be more of a general issue than axolotl's as others above receive it despite not using axolotl.

The problem is, I don't even know "what I need" as far as configs go.  Here's my goals:

Train a 3B model
... on my Linux box on GPU #0 only (CUDA_VISIBLE_DEVICES=0),
... using a dataset I built, in alpaca json format
... ideally using as little memory in training as possible (not critical here for this tiny 3B model, but important for bigger future models)
... ideally as fast as possible,
... and ideally yielding a 4-bit quantized GPTQ model

So I'm trying to follow the example and the "guide" on the axolotl project page, but there's tons of parameters and config options, and everything seems to lead down a "that's broken, with an obscure error message" road.  :(
So should I be commenting out some line of code?  Which lines of code? There's a whole stacktrace:
Traceback (most recent call last):
File "/home/username/axolotl/src/axolotl/utils/models.py", line 194, in load_model
model, _ = load_llama_model_4bit_low_ram(
File "/home/username/.local/lib/python3.10/site-packages/alpaca_lora_4bit/autograd_4bit.py", line 249, in load_llama_model_4bit_low_ram
model = accelerate.load_checkpoint_and_dispatch(
File "/home/username/.local/lib/python3.10/site-packages/accelerate/big_modeling.py", line 486, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "/home/username/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 1116, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
File "/home/username/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.")
ValueError: Autograd4bitQuantLinear() does not have a parameter or a buffer named weight.
Traceback (most recent call last):
File "/home/username/axolotl/src/axolotl/utils/models.py", line 194, in load_model
model, _ = load_llama_model_4bit_low_ram(
File "/home/username/.local/lib/python3.10/site-packages/alpaca_lora_4bit/autograd_4bit.py", line 249, in load_llama_model_4bit_low_ram
model = accelerate.load_checkpoint_and_dispatch(
File "/home/username/.local/lib/python3.10/site-packages/accelerate/big_modeling.py", line 486, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "/home/username/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 1116, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
File "/home/username/.local/lib/python3.10/site-packages/accelerate/utils/modeling.py", line 149, in set_module_tensor_to_device
raise ValueError(f"{module} does not have a parameter or a buffer named {tensor_name}.")
ValueError: Autograd4bitQuantLinear() does not have a parameter or a buffer named weight.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/username/axolotl/scripts/finetune.py", line 352, in
fire.Fire(train)
File "/home/username/.local/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/username/.local/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/home/username/.local/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/home/username/axolotl/scripts/finetune.py", line 251, in train
model, peft_config = load_model(
File "/home/username/axolotl/src/axolotl/utils/models.py", line 290, in load_model
model = AutoModelForCausalLM.from_pretrained(
File "/home/username/.local/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 467, in from_pretrained
return model_class.from_pretrained(
File "/home/username/.local/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2611, in from_pretrained
model = cls(config, *model_args, **model_kwargs)
TypeError: LlamaForCausalLM.init() got an unexpected keyword argument 'load_in_4bit'
Traceback (most recent call last):
File "/home/username/.local/bin/accelerate", line 8, in
sys.exit(main())
File "/home/username/.local/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)
File "/home/username/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 941, in launch_command
simple_launcher(args)
File "/home/username/.local/lib/python3.10/site-packages/accelerate/commands/launch.py", line 603, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'scripts/finetune.py', 'summarize.yaml']' returned non-zero exit status 1.

But this is getting off topic - I'll start a new thread  :)

So I've been doing more testing, and regardless of peft / gptq install status, doing the transformers==4.29.2 downgrade always leads directly into:
TypeError: LlamaForCausalLM.init() got an unexpected keyword argument 'load_in_4bit'
Indeed, while I can train the example lora yml (not the example model yml) if I do the right combination of install steps (no deepspeed, no torch compiling, no low-bit floating point config, and the peft-pull rather than the gptq requirements install), once I do the downgrade, neither loras nor models can be trained - both hit the above error.

TypeError: LlamaForCausalLM.init() got an unexpected keyword argument 'load_in_4bit'

Hello @enn-nafnlaus , sorry for late reply. I forgot to mention which lines. It's this mainly this line here https://github.com/OpenAccess-AI-Collective/axolotl/blob/b9b7d4ce9292739d7bd3b6113e54786f45db7462/src/axolotl/utils/models.py#L213
Do note: commenting this out would not allow QLORA.

I came across this again when I needed to use newer version of transformers. The thing that fixed it now is using latest accelerate.
transformers==4.31.0
accelerate==0.21.0


simp_le (and security.acme) is broken because certbot is too new

Issue description
Collecting acme<0.6,>=0.4.1 (from simp-le==0)
  Could not find a version that satisfies the requirement acme<0.6,>=0.4.1 (from simp-le==0) (from versions: )

Steps to reproduce
nix-build -A simp_le .

Technical details
nixpkgs: cc41cb30e33a071e4e5e86d277579fc02d0bf7de

Oh, sorry, this was fixed on master with cc41cb30e33a071e4e5e86d277579fc02d0bf7de and I haven't noticed.


Add gulp-csslint reporting

Reading through linters & curious how valuable csslint would be on this project, see https://github.com/lazd/gulp-csslint

getting the error linked below when implementing this. seems to have an issue understanding new css features like variables
https://stackoverflow.com/questions/22747721/expecting-rbrace-what-is-css-lint-talking-about

I realize it's not a big need anymore because stylelint is more supported & has overlapping features. We might want to look at which stylelint plugins to look into https://stylelint.io/user-guide/plugins/


[Question] Integrating angular-formio with rest-services and database

Hi Community,
I have to integrate this with my application.
I want a full guide to do this without using portal.formio
I want to create form and i can store the json components in database.
I want to render the form from json component from database.
I want to display the submission data and want to edit or delete the submission data.
Is this possible? please guide me

+1

Closing this thread as it is outdated. Please re-open if it is still relevant. Thank you for your contribution!


feat: Blobstream errors

[x] issue #26
[x] follows contribution guide
[x] code change includes tests
[x] breaking change

Contract errors and implementation of the function using it

If necessary I can also finish everything related to the general functions implementation (tests + misssing functions), but I don't know really if I should be doing it here.

If necessary I can also finish everything related to the general functions implementation (tests + misssing functions), but I don't know if I should be doing it here.

I'll open a new one and assign to you.

That's weird I wonder why the formatter doesn't like that comment indent. You can keep the left formatting and we will clean up in subsequent work.

https://github.com/all-contributors please add @JordyRo1 for code

@all-contributors please add @JordyRo1 for code

@all-contributors add @JordyRo1 code

@all-contributors add @JordyRo1 for code

@all-contributors add JordyRo1 for code


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-archive-test-target/pull/1068 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-archive-test-target.git

Docs Build status updates of commit 52beef2:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: RepositoryUnauthorizedWithSourceCommit] Git data access to 'https://github.com/opstest2/docs-archive-test-source' (branch: Release_Archive_main_2022-02-22-01-17-15) is not authorized.. It looks like this might be the first time user: v-alji submitted a pull request to this repo. Before user can run builds in a pull request, he/she can enable 'v-alji' GitHub account in Docs by visiting Docs Portal (https://ops.microsoft.com). After enabling his/her account (v-alji), close and then reopen this pull request to start another build.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


How the models construct?

Hi,
Your work is very interesting and thanks for sharing the codes!  However, I cannot find the training codes. Can you share with me? I would like to ask how the FR and NR models construct, using what kind of framework, like caffe, tensorflow and so on?

Currently, I did not have the time to clean up the training code for release. But the models, as defined in fr_model.py and nr_model.py, contain anything necessary for training using the chainer framework and your own data loading code. All details regarding training should be documented in the paper.

Hi @dmaniry. Great work!! I am interested in the implementation of NR IQA method, however in your code, you have only defined the forward pass and not any backward pass. Can you provide a snippet of your training code?

@dmaniry , I've been trying to implement the training using the details provided in the paper, but the model doesn't converge at all. Would be VERY helpful if you could put up the training code so we can have a look at it and use it for our implementation!
Thanks, really appreciate it!


Utilize GithubCheckRun.app property

Requires packit/ogr#655

I've built & pushed the packit-service-validation image with ogr installed from git and will merge this PR once new ogr is released and pushed to F35 stable (because the image installs it from there).


Added bower support to readme

Since people (like me) will notice that there is a bower.json file in the repository, they might try to install the file via bower and wonder why it does not work. So I thought it might be better to inform them.

Yeah, I think there needs to be a 0.0.11 tag based on origin/master that's organized correctly - it looks like #12 was merged into master, but that actually contains a compiled version of player.js that was built against src/ files on the 0.0.10 tag.  My idea behind the PR was that:

Player.js would be published as a bower component by Embedly/@screeley
origin/master would be maintained without a dist/ directory
Periodically and at the discretion of the repo owner a new tag would be created
dist/ would be generated and committed to that tag
bower update for any bower consumers
Do awesome player.js things

Thanks,
-Eric

I threw it up on bower today and bumped the repo to 0.0.11. While the tag dist/ upload would be better, I kept it simple and just committed dist/ into master.
Sorry for being slow on this.


Cannot connect to remote existing kernel on macOS Catalina

Issue Report Checklist

[X] Searched the issues page for similar reports
[X] Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
[X] Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
[x] Could not reproduce inside jupyter qtconsole (if console-related)
[X] Tried basic troubleshooting (if a bug/error)

[X] Restarted Spyder
[X] Reset preferences with spyder --reset
[X] Reinstalled the latest version of Anaconda
[X] Tried the other applicable steps from the Troubleshooting Guide

[X] Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description
Spyder fails to connect to remote kernel running on Ubuntu from MacOS.
I'm able to connect to the kernel using this recipe so I do not think this is a problem with my setup.
What steps reproduce the problem?

Follow the steps in the doc

What is the expected output? What do you see instead?
Could not open ssh tunnel. The error was: Could not connect to remote host

Versions

macOS Catalina (client)

Spyder version: 3.3.6
Python version: 3.7.5
Qt version: 5.9.7
PyQt version: 5.9.2
Operating System name/version: macOS Catalina 10.15.1

Ubuntu (server)

Spyder version: 3.3.6
Python version: 3.7.5
Qt version: 5.9.7
PyQt version: 5.9.2
Operating System name/version: Ubuntu desktop 18.04 LTS

Dependencies

IPython >=4.0     :  7.9.0 (OK)
cython >=0.21     :  0.29.14 (OK)
jedi >=0.9.0      :  0.15.1 (OK)
matplotlib >=2.0.0:  3.1.1 (OK)
nbconvert >=4.0   :  5.6.1 (OK)
numpy >=1.7       :  1.17.4 (OK)
pandas >=0.13.1   :  0.25.3 (OK)
psutil >=0.3      :  5.6.5 (OK)
pycodestyle >=2.3 :  2.5.0 (OK)
pyflakes >=0.6.0  :  2.1.1 (OK)
pygments >=2.0    :  2.4.2 (OK)
pylint >=0.25     :  2.4.4 (OK)
qtconsole >=4.2.0 :  4.6.0 (OK)
rope >=0.9.4      :  0.14.0 (OK)
sphinx >=0.6.6    :  2.2.1 (OK)
sympy >=0.7.3     :  1.4 (OK)

Sorry but we don't the means to debug this. If the IPython recipe is working for you, then you'll have to use that instead.

Feel free to keep this closed if you want but I think you’re ignoring a sign of broken functionality. If I have to use a console outside of the IDE to run my code why use Spyder at all?

Feel free to keep this closed if you want but I think you’re ignoring a sign of broken functionality. If I have to use a console outside of the IDE to run my code why use Spyder at all?

That's your choice. The functionality of running in remote kernels runs well in most cases, but we can't cover all OS host/guest combinations.

Hi,
I  had a very similar problem. I tried to connect to a remote kernel. Client and Server are both linux system (Debian 10 and CentOS7) and both use the same software versions (python 3.7.3, spyder 4.0.1, spyder_kernels 1.8.1, IPython 7.13.0).
I create on a server the remote kernel with python -m spyder_kernels.console --debug -f kernel-6787.json to use an old configuration.
On the client i follow the normal workflow and go to the menu and choose the entry "connect to ..." and take the copied kernel file.
Then happened nothing and on the internal console is written out, that "WARNING:traitlets:kernel died: 3.00736"
As a work around I use the description here, to set up the ssh tunnel manual and than connect to remote kernel. That is working.
Is there a possibility to debug the problem?

This isn’t really a solution, but I ended up just setting up a Jupyter notebook server on my Ubuntu machine in order to do my work.
Again, not a solution to the problem but it worked well for my situation.


refresh token user id not saved properly?

I am using the mongoose models from the example and have the following problem:
I authenticate with grant_type password, I receive a refresh token. Once the access token is expired I make a request with the refresh token, but the new access token/refresh token models don't have the "userId" property in the database, so the authentication stops working.
It works when I change line 264 in grant.js from:
self.user = refreshToken.userId; || { id: refreshToken.userId };

to
self.user = refreshToken.userId;

But I am guessing that's there for a reason.. any ideas?

I assume this will be due to the model implementation, have you modified it in anyway from the one in the examples?
If not I will have a look through the one in the examples to check

I have, but only a few minor changes. I'll look to see if I can find the problem there.

The only difference in the model from the example is that I use my own client model in getUser:
model.getUser = function (username, password, callback) {
    Client.findOne({emailAddress: username}, function(err, client) {
        if (err) {
            return callback(err);
        }

        if (client && client.validPassword(password)) {
            return callback(null, client._id));
        }
        return callback(null, null);
});

For now I have fixed it by changing client._id to client._id.toString() and changing the saveRefreshToken and saveAccessToken methods to the following:
model.saveAccessToken = function (token, clientId, expires, userId, callback) {
    if (typeof userId.id == "string") {
        userId = userId.id;
    }

    var accessToken = new OAuthAccessTokensModel({
        accessToken: token,
        clientId: clientId,
        userId: userId,
        expires: expires
    });
    accessToken.save(callback);
};

model.saveRefreshToken = function (token, clientId, expires, userId, callback) {
    if (typeof userId.id == "string") {
        userId = userId.id;
    }

    var refreshToken = new OAuthRefreshTokensModel({
        refreshToken: token,
        clientId: clientId,
        userId: userId,
        expires: expires
    });
    refreshToken.save(callback);
};

~ here. The lib calls to saveAccessToken with an inconsistent userId model.
With grant_type password, userId is "JuanCarlos".
With grant_type authorization_code, userId is { id: "JuanCarlos" }


Add open and merged PRs as a secondary metric to ProjectIssues

Note "merged" PRs should be counted and not "closed" PRs since that would be a redundant metrics since that code may not have ever been merged and if it has then that's counted with merged PRs

Note "merged" PRs should be counted and not "closed" PRs since that would be a redundant metrics since that code may not have ever been merged and if it has then that's counted with merged PRs

This issue overall was implemented but this note in the description was not considered.
Currently the logic will show all closed PRs but it is possible to close a PR without merging the code so that wouldn't be useful to count in these metrics. Will tackle this in a separate issue


Replace gsl::byte/span with std

This is a rather trivial changeset. Now that these two are present in the
std namespace there's no reason for us to continue using the gsl ones.
Additionally this ensures future compatibility with other 3rd party libraries.

For future reviewers: Leonard is not lying when he says it's all mechanical. If you review only one file, make it DispatchTypes.cpp


fix temperature reading

At the moment the temperature is set from what is measured at the UV2 channel ,             CCDitem["temperature"] = HTR8A  # ,
see get_temperature.py. This should be made in a better way.

Duplicate of #65


Adding karaoke feature

🚀 Feature
Using Web Audio API , I would like add on  karaoke filter  to remove the voice from a song.
Drag and drop a music file (mp3) and sing it !!

/assign


Added more info to readme

Added reminder to update api.projectId in studio/sanity.json when running studio locally
Added instruction on how to enable preview mode from Sanity Studio

Thanks @hyldmo !
If you follow the instructions for initializing Sanity (manual setup) the project ID in sanity.json should update automatically, so I don't think we need this note added.
As for the additional verbiage around previews... I also think this may be better suited called out in the setup details. I think the features list should remain short and concise, and any details about how to set something up should exist elsewhere.
Hope that makes sense! Let me know if you have any additional feedback though!


Bidirectional LSTM Added

Related issue : #300

@pratishthaa Thank you for merging.Kindly add the labels


modular basins: Ridge Basin

voxet data:
CVMHB-Ridge-Basin.zip

points:

https://github.com/SCECcode/cvmhbn


use rcedit to embed icon and version information into pwsh.exe

pwsh.exe today doesn't contain file version information and the icon is only associated with the shortcut file and not the exe
Fix is to use rcedit to embed:

icon
product version
file version
product name
copyright

Fix https://github.com/PowerShell/PowerShell/issues/2883
Fix https://github.com/PowerShell/PowerShell/issues/5166
Fix https://github.com/PowerShell/PowerShell/issues/5034

This will fix #5034 as well.

It seems CoreFX use System.Reflection.Metadata to read FileVersionInfo. Perhaps we can use the API write meta data too.
CoreFX implementation

@iSazonov I think we can move to that later particularly if we expose cmdlets for it

@SteveL-MSFT Do you want to exclude Unix from the PR?

@iSazonov using the file command, the executable is a native executable whereas the .dll assemblies are PE32+ format which supports the embedded resources.  I don't think we can support this for non-Windows executables.

@SteveL-MSFT Thanks for clarify. I see Unix haven't a standard for this.


iOS 6 support in latest version broken

The latest version crashes when ran on iOS 6. The reason is because of line 122 in RDVTabBarController.m.
[self setNeedsStatusBarAppearanceUpdate];

Needs changed to:
if ([self respondsToSelector:@selector(setNeedsStatusBarAppearanceUpdate)])
{
        [self setNeedsStatusBarAppearanceUpdate];
}

me too


feat: responsify history funding payments table

This PR:

resolves #753
figma design

Screenshot:

Looks good ✅
Just one question about sorting... how is it realised now? If I had 2 positions opened for 2 different pairs I'd expect the funding payments to be displayed more or less like that:

funding for pair A
funding for pair B
funding for pair A
funding for pair B

But what I actually saw was:

@terencetang9 yeah, shouldn't block. Let's create an enhancement issue to sort by timestamp and if that's equal, then sort by ticker?

@Tofel yes will do!


ci: merge staging to master

This is an automatic PR generated by the pipeline CI/CD. This will be automatically fast-forward merged if successful.

Pipeline Attempt on 914224897 for eabdced1c004580457bd34e3d770c07e8ef661fd
https://gitlab.com/MatrixAI/open-source/js-workers/-/pipelines/914224897

Pipeline Attempt on 914225267 for eabdced1c004580457bd34e3d770c07e8ef661fd
https://gitlab.com/MatrixAI/open-source/js-workers/-/pipelines/914225267

Pipeline Attempt on 914228285 for d91ce6f1871c04b28382738bb571acc5f1e58be0
https://gitlab.com/MatrixAI/open-source/js-workers/-/pipelines/914228285

Pipeline Attempt on 914228287 for d91ce6f1871c04b28382738bb571acc5f1e58be0
https://gitlab.com/MatrixAI/open-source/js-workers/-/pipelines/914228287

Pipeline Succeeded on 914228285 for d91ce6f1871c04b28382738bb571acc5f1e58be0
https://gitlab.com/MatrixAI/open-source/js-workers/-/pipelines/914228285


🐛 Update BMO dependency for root device hints

What this PR does / why we need it:
This PR updates BMO to allow the use of root device hints https://github.com/metal3-io/baremetal-operator/pull/495

/test-v1a4-integration

/lgtm


Rewards tab is directing to GoodMarket (Webapp, iOS, iPhone8)

@LiavGut couldnt reproduce. closing.


Documentation Index

Summary:
This update introduces a central index for documentation and improves the overall structure of the docs folder, making it easier for users to navigate and access information.
Key Changes:

New README as docs index: Added a README file to the docs folder, serving as a central index for all documentation pages.
Documentation restructure: Moved existing docs pages into a new /docs/pages subfolder for better organization.
Logo relocation: Moved logo from the repo's root into docs and used raw.githubusercontent.com URL. It's cleaner. Using a direct URL will also allow displaying the logo on the PyPI page (now it's missing). Note: the link to the logo uses the URL's main branch, so it will only work once we merge the PR.
Aana App Template Repo: Added link to aana app template repo.
Documentation fixes: Corrected various minor issues and inaccuracies in the existing documentation.

Can you also add https://github.com/mobiusml/aana_sdk/wiki/Cluster-Setup ( maybe move it out of wiki to docs ) and add a section on scaling up the SDK ?
There are ways to monitor Ray cluster, and is not apparent. We should probably talk about it somewhere ( not in detail during this version )
We should link the videos etc. here. But it can be a part of a different PR later in future.

Can you also add https://github.com/mobiusml/aana_sdk/wiki/Cluster-Setup ( maybe move it out of wiki to docs ) and add a section on scaling up the SDK ?

There are ways to monitor Ray cluster, and is not apparent. We should probably talk about it somewhere ( not in detail during this version )

We should link the videos etc. here. But it can be a part of a different PR later in future.

We should also mention about the template. Possibly in the development -> development guide.

The seperation of projects and SDK is not apparent.

I already mentioned the template.

The seperation of projects and SDK is not apparent.

What do you mean and what do you want me to do about it?

Let us first move existing projects out and then a subsection in getting
started section to make it clear
On Fri 12. Jul 2024 at 12:06, Aleksandr Movchan @.***>
wrote:

Can you also add
https://github.com/mobiusml/aana_sdk/wiki/Cluster-Setup ( maybe move
it out of wiki to docs ) and add a section on scaling up the SDK ?
2.
  There are ways to monitor Ray cluster, and is not apparent. We
  should probably talk about it somewhere ( not in detail during this version
  )
  3.

  We should link the videos etc. here. But it can be a part of a
  different PR later in future.
  4.

  We should also mention about the template. Possibly in the
  development -> development guide.
  5.

  The seperation of projects and SDK is not apparent.

I already mentioned the template.
The seperation of projects and SDK is not apparent.
What do you mean and what do you want me to do about it?
—
Reply to this email directly, view it on GitHub
https://github.com/mobiusml/aana_sdk/pull/135#issuecomment-2225249695,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJL5SMY6ILMDW2MT5HMY7DZL6TDNAVCNFSM6AAAAABKYSXXSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVGI2DSNRZGU
.
You are receiving this because your review was requested.Message ID:
@.***>

Let us first move existing projects out and then a subsection in getting started section to make it clear
…
On Fri 12. Jul 2024 at 12:06, Aleksandr Movchan @.> wrote: 1. Can you also add https://github.com/mobiusml/aana_sdk/wiki/Cluster-Setup ( maybe move it out of wiki to docs ) and add a section on scaling up the SDK ? 2. There are ways to monitor Ray cluster, and is not apparent. We should probably talk about it somewhere ( not in detail during this version ) 3. We should link the videos etc. here. But it can be a part of a different PR later in future. 4. We should also mention about the template. Possibly in the development -> development guide. 5. The seperation of projects and SDK is not apparent. I already mentioned the template. The seperation of projects and SDK is not apparent. What do you mean and what do you want me to do about it? — Reply to this email directly, view it on GitHub <#135 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJL5SMY6ILMDW2MT5HMY7DZL6TDNAVCNFSM6AAAAABKYSXXSGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRVGI2DSNRZGU . You are receiving this because your review was requested.Message ID: @.>

Let us first move existing projects out

I'm working on it but it will be a separate PR. Is there anything you want me to do about it in this PR?

This is good to go


Major Windows updates continually uninstall the service

After every major Windows update e.g. 8.0 > 8.1, 8.1 > 10, or now, 10 to the new build, the DNSCrypt proxy service will always be removed, leaving people with broken internet connections.
I assume the solution to this would be signing the Windows service, which should be easy now with LetsEncrypt?

Signing the binaries may solve this. But TLS certificates and Windows Developer certificates are different and incompatible things.
And Letsencrypt will only provide the former. So, this is unlikely to change anytime soon.

It's disappointing that you closed this, as what I stated was not a fact, it was a guess.
The problem here is that DNSCrypt is no longer viable on Windows with this issue, unless you're only installing it on your own machine. I now have to guide people through fixing their internet connections since the new Windows 10 update will break it for them.
Security should not be a hassle, it should be simple and effort free. Unfortunately here the simple and effort free path is telling them to reset back to using their ISPs DNS.

Are you willing to pay for the certificate?

Before asking that it has to be asked: Are we 100% sure this is the solution to the problem? If so, I'm sure people would be willing to crowdfund/donate. I would.

Here's a cheap one if this is the problem:
https://en.sklep.unizeto.pl/data-safety/code-signing-certificates/open-source-code-signing.html
The company is in the list of Certification Authorities who are members of the Windows Root Certificate Program.

@funkydude inside this msi are signed files: https://github.com/bitbeans/SimpleDnsCrypt/releases

@bitbeans does it stay after major Windows build updates?

i have not tested this scenario, but i don`t see any problems

@bitbeans Do you not run Windows 10? It is reproducible by updating to the latest build released this past week.
@jedisct1 What do you think about a 14 euro cert?

@bitbeans Well I'm trying SimpleDNSCrypt in the hopes that the signed service won't be removed. However, my main issue is when I try to use Cisco OpenDNS, then close the application, then open it, it states "Cisco OpenDNS Backwards compatability". This is not the option I selected, and I did not have this issue when performing the basic install as a service.
Also when I tick the box to enable it on my WiFi adaptor (which is disabled in Windows, and marked as red in your UI), the option fails to take effect, and is unticked when I next launch the application.

@funkydude maybe it`s the wrong project to discuss this :)

Well I'm trying SimpleDNSCrypt in the hopes that the signed service won't be removed. However, my main issue is when I try to use Cisco OpenDNS, then close the application, then open it, it states "Cisco OpenDNS Backwards compatability". This is not the option I selected, and I did not have this issue when performing the basic install as a service.

fixed in the latest commit (ty).

Also when I tick the box to enable it on my WiFi adaptor (which is disabled in Windows, and marked as red in your UI), the option fails to take effect, and is unticked when I next launch the application.

fixed in the latest commit (ty).

Where is the configuration file located? Or is it modifying the registry entries?

There is no configuration file at the moment. The application only needs the dnscrypt-proxy registry entry.

Another question is why is libsodium installed and run from the temp directory and not in the program files directory?

This is because i use https://github.com/Fody/Costura to merge the assemblies.

Sorry I made the mistake of looking at the DLLs for the GUI application, which doesn't launch when Windows loads anyway, so it doesn't matter that it loads a temp file (assuming it's created every time the GUI launches).
I've updated, thanks, but I have 2 more issues. The following DLLs that are loaded by the proxy service are not compiled with the ASLR flag.
libwinpthread-1.dll
libgcc_s_dw2-1.dll
Fixing that, and assuming Windows won't remove the service since it's signed, I can (hopefully) start recommending DNSCrypt again.

@jedisct1 do you need the 14 euros donated?
@bitbeans do you plan on resolving above issues?

@bitbeans It seems as of 0.2.9 the IP and adaptor issues are resolved. However, the DLLs are still not compiled with ASLR, which is the last remaining (security) issue.

@jedisct1 why is there still no cert?
@jedisct1 @bitbeans ASLR is still a problem in the latest release.

@funkydude i do not recompile them. i use always the official release version.

@bitbeans @jedisct1 I tried to fetch version info from both DLLs but could not, I just found "build time" with very concerning attributes of Fri Mar 14 20:17:06 2003 for libgcc_s_dw2-1.dll and Fri Feb 05 18:21:39 1999 for libwinpthread-1.dll
libsodium correctly has a 2016 date, what's going on?

It's the gcc runtime library. It has nothing to do with the dnscrypt-proxy source code.

@jedisct1 doesn't answer why you appear to be using an ancient version.

That's what the latest version on MinGW package in MSYS2 ships with.

@jedisct1 I downloaded from https://msys2.github.io/
It contains a msys-gcc_s-1.dll
It doesn't contain winpthread...

...then you have to install MinGW.


boltdb: Improve node dumps

This patch adds better debugging logs to boltdb.  I found it useful.  This is a rebased version of the patch I submitted to the boltdb project at https://github.com/boltdb/bolt/pull/597.
To use it when bolt is running in a background process the log needs to be written to a file.  Here's how I do that for the tests:
--- a/etcdmain/main.go
+++ b/etcdmain/main.go
@@ -16,10 +16,19 @@ package etcdmain

 import (
 	"fmt"
+	"log"
 	"os"
 )

 func Main() {
+	logfile, e := os.OpenFile("logfile", os.O_RDWR | os.O_CREATE | os.O_APPEND, 0666)
+	if e != nil {
+		panic("logfile")
+	}
+	log.SetOutput(logfile)
+	log.SetFlags(log.Lshortfile)
+	log.Println("Main >")
+
 	checkSupportArch()

 	if len(os.Args) > 1 {

Can't merge this unless it's in upstream since it's a vendored 3rd party package.

For anyone interested, I'll keep this in my master branch.


import IVisualHost to visual.ts

In the developing-the-visual-elements section, IVisualHost needs to be imported. Otherwise the compiler gives the error:
./src/visual.ts
[tsl] ERROR in C:\Users\Andy\pbiviz\circleCard\src\visual.ts(40,19)
      TS2304: Cannot find name 'IVisualHost'.

@andyparkerson : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@maggiesMSFT : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@zBritva can you please confirm that these changes are correct? Thanks.

@KesemSharabi It's correct, but issue already was fixed: https://docs.microsoft.com/en-us/power-bi/developer/visuals/custom-visual-develop-tutorial#developing-the-visual-elements

@KesemSharabi - if the issue is already addressed, let us know if you prefer to close this PR by entering #please-close in a comment.  thanks.


does the function adjustSampleClockRelativeDelay delay external clock

AB#2323114

@Achuan-2 please ask general questions in the Discussions tab at https://github.com/ni/grpc-device/discussions - this way your question will be visible to others in the community who might also be able to help. We prefer to use GitHub Issues for tracking bugs or requests for changes in the code or related documentation.

Since this has been open for a while, I'm not asking to close this and re-post in the discussion forum. Keeping this issue open until we can answer this, for this instance

ok


config option to bind pleroma webendpoint on all interfaces

Hi,
i dunno if it helps or if you want it in your readme, but pleroma recently changed their default ip binding -> https://git.pleroma.social/pleroma/pleroma/issues/614
so i had overwrite their changes and bind pleroma back on <IP_ADDRESS> cause without that my pleroma wouldn't be reachable through the nginx proxy on the host maschine (i think it still works within the same docker network, via docker-compose etc)

Thanks


Question, not an issue.

I was hoping your moonmoji was going to give me the colored emoji pictures that are displayed on NPM. I'm fairly new to this sort of thing so it's quite possible that I am missing something or that I misunderstood. Would you kindly just verify for me whether or not there are colored emoji icons included or just the black and white ones that I seem to be getting. Many thanks. :-)

they should be colored:


[BUG] CI should validate built plugin file structure

What is the bug?
If the file structure is incorrect (missing License for example) the CI passes.
How can one reproduce the bug?
Steps to reproduce the behavior:

Remove file extention of License
Build project
File is missing from folder
Commit changes in new branch and see that CI passes.

What is the expected behavior?
CI validates files in plugin. Fails on any missing files.

@lukas-vlcek if you are up for it let me know:)

@AmiStrn I can take it. Feel free to assign it to me.


Do not attempt LB plumbing until sandbox is ready

Also do not log error messages when adding a destination and it already
exists. This can happen because of duplicate gossip notifications.
Signed-off-by: Jana Radhakrishnan<EMAIL_ADDRESS>

LGTM


[VSCode] live-share should double click to join / disconnect from session

Hi Vscode dev here,
Platform independent, though I use OS X

VSCode, install live share extension
Open the live share view, single click on the start collaboration session
Notice how the live share session is immediatly starting

This should be only be done on double click, or when the element has focus + enter.
Otherwise there is no way for me to click on the elment to focus it and navigate around.
Especially since this is a "heavy" operation
Also this is not consistent with how other trees in vscode are behaving.

@isidorn Thanks for the feedback! Does the TreeItem API support specifying a command to run on double-click (the command property seems specific to single-click)?
We actually thought this behavior was idiomatic for VS Code, and modeled these links roughly after the default tree experience for the Azure extensions. So if we were wrong about that UX, then we'd definitely want to change it.

@sandy081 would know better whetever we support extensions to contribute commands on double click. Looking at the API I would say no.
@lostintangent I was comparing it to how the vscode core behaves, I was not sure if we opened this customisation for extensions once I filled this issue.
If all extensions behave this way than I would say you are aligned, even thuogh I personally do not liek the experience.

Double click is not supported - https://github.com/Microsoft/vscode/issues/39601#issuecomment-398751109

I’m going to close this issue since as @sandy081
mentioned, double-click isn’t currently supported. Additionally, we haven’t heard any feedback about this from users, so we’ll wait to hear more feedback before pursuing any other changes. Thanks!


Allowed OAuth Scopes

The values Cognito and profile are not working for OAuth Scopes

Hi @marcschroeter!
I came across this as well. In fact I have just submitted a Pull Request to @rosberglinhares. Check it out!
AWS docs are wrong. If your are looking at:
https://docs.aws.amazon.com/cognito-user-identity-pools/latest/APIReference/API_UpdateUserPoolClient.html#CognitoUserPools-UpdateUserPoolClient-request-AllowedOAuthScopes
There it mentions the current supported values:
"phone", "email", "openid", and "Cognito"
But in reality, the current supported values are:
"phone", "email", "openid", "aws.cognito.signin.user.admin", "profile"
Your problem is using "Cognito" instead of "aws.cognito.signin.user.admin". Let me know if this helps you!


Safely test for optional "type" field in poetry lock

The "source" field in a poetry lock file can optionally have a "type"
field. However, in cases where an alternative "simple" index is offered,
e.g., artifactory, this type field is not used. This patch makes
checking this type field safe by returning a None in the case that the
field doesn't exist.
Fixes: https://github.com/dephell/dephell/issues/437

Do you have any feedback @orsinium ? Thanks!

Great! Thank you :)


Integrate kafka log appender changes and align promotion request styles

Promotion requests used to bypass the request threadpool for sync calls, which also bypassed the load check in the threadpool. This change uses the Future<> from those async versions of the call and simply performs a Future.get(), returning the result afterward. That will force all requests to go through the threadpool, which subjects them all to the same capacity check.
Also, incorporating the kafka log appender so we can start learning to send logs to Kafka.

This supercedes #1126

retest this please


Update quickstart-powershell.md

Fix typos.

@alexandair : Thanks for your contribution to the Azure documentation! The author, @maertendmsft, has been notified to review your proposed change.

@alexandair, thank you for your feedback and helping to improve the Cloud Shell documentation :)

@maertendMSFT Could you merge this PR?


[Enhancement] generalize tbb's thread_pool_base

Why?
I would like to use the library with another thread pool than tbb. This PR contains a proposed implementation for this enhancement and solves some issues.
Current Issues
For this purpose the tbbexec::thread_pool_base looks like a perfect abstraction. But currently it has two issues:

It is in the tbb_thread_pool.hpp, therefore it has a dependency with tbb.
It has a friend declaration friend struct DerivedPoolType::tbb_thread_pool::scheduler; where the tbb_thread_pool actually bounding it to the tbb implementation and it is also unnecessary. I.e.: friend struct DerivedPoolType::scheduler; would be also good.

Proposal
My proposal to having a more generic pool hierarchy in the stdexec. Currently it looks like:
 tbbexec
    | - tbb_thread_pool.hpp

I would go with adding 3 implementation for 3rd party thread pools (3 because I like the Rule Of Three):
 execpools
  | - boost
  |    | - boost_thread_pool.hpp
  | - taskflow
  |    | - taskflow_thread_pool.hpp
  | - tbb
  |    | - tbb_thread_pool.hpp
  | - thread_pool_base.hpp

I choose these 3 3rd-party library, because I think these are the most popular threading libraries.
About the PR

I've tried to organize the commit properly, making easier to follow the changes.
I used the llvm18-cuda12.4 dev-container to test the changes.
I formatted the changed files with clang-format
Added two new cmake option: STDEXEC_ENABLE_BOOST, STDEXEC_ENABLE_TASK_FLOW. These should control the dependencies and fetches those when it is necessary.

I've rebased the PR and also implemented @mathisloge comment.
I have a bit of a feeling to using the adding the standalone ASIO adds a bit of a complexity to the whole architecture and I'm up for reverting if it causes maintenance issues. The source of the complexity:

Standalone Asio isn't based on CMake, thus importing it even with CPM is not so simple. (Needed to add a cmake script)
Because the asio implementation depends on the configuration a generated header file needed to be added (asio_config.h.in). Though the generated file is not part of the source tree It ads another thing what needs to be maintained.

If you have any better idea to integrate the standalone asio let me know.

apologies for the delay. i'll get to this soon i hope.

/ok to test

I've pushed some quick update on it:

removing the copy-paste errors on the base thread pool (tbb references, dead code)
on and transform got deprecated, I replaced these function usage on the tests
adding back the tbbexec header. @ericniebler I've also added a warning to the file. My idea is that everyone who uses the old header got a notification about the changes with this warning. I mean I like to be backward compatible but it is also nice if there is a way to remove this later on and doesn't leave in the repo forwever.

@ericniebler I think I can see the way how to use the same scheduler and sender in static_thread_pool and the other pools. So, these can have a common base class. I'm a bit worried about the details what I can't see right now (the possible tiny differences between static- and tbb/base- thread_pool, but theoretically it should work. I'm working on it to make this happen really soon.
@BenFrantzDale I'm gonna add as an example the AnyThreadPool I like the idea of having more example for the usage!

think I can see the way how to use the same scheduler and sender in static_thread_pool and the other pools. [...] I'm working on it to make this happen really soon.

no rush, that can be a separate PR.

/ok to test

thank you!


MLHR-1755 #resolve Convert all demos in malhar to generate file with apa extension instead of DTApp-name-version.jar

test this please


Small updates to NERSC installation tutorial

Asking to add these:
module load python/2.7-anaconda-4.4
pip install --user healpy

Sorry for the massive delay @nsevilla . All done now, thanks a lot!


Issue with visualization

I am getting an error with the visualization step.
`Querying chunk 56
Finished
Pass returnall=TRUE to return lists of duplicate or missing query terms.
Error in '$<-.data.frame'('tmp', "lab", value = c("Lcn2", "Cd24a", "Sprr1a",  :
replacement has 55422 rows, data has 55421
Calls: plot.volcano.heatmap -> EnhancedVolcano -> $<- -> $<-.data.frame
Execution halted
`
I am also getting the warning
`Loading required package: ggrepel
Warning message:
In readLines(con) :
incomplete final line found on 'configs/config_main.yaml'
Querying chunk 1
Querying chunk 2
`
Any help with these? Am I missing a package in R?
I also noticed it's not possible to use the main.py to run just the visualization without redoing the DEA, running it with DEA set to returns
Which mapping reference will be used?  genome Is DEA requred?  False Is visualization requred?  True Please double check the information above Do you want to continue? (y/n) y Start RASflow on project: project Trimming is not required Start mapping using  genome  as reference! Building DAG of jobs... Nothing to be done.

Thank you for reporting!
About your question of "it's not possible to use the main.py to run just the visualization without redoing the DEA", it's more about the logic behind the workflow: visualization is to visualize results of DEA, so visualization cannot happen without finishing DEA. So if you want to do visualization, DEA has to be set "yes". But if you have already finished DEA, and now only want to do visualization, no worry about "redoing" the DEA, because when Snakemake finds that the output files of DEA are already there, it will skip DEA.
About the errors, missing a package won't be the reason. Because once you set up the environment with env.yaml, all required packages are installed.
For "incomplete final line" in yaml file, I would guess that when you modified config_main.yaml, you accidentally removed some lines. Try adding an empty line in the end of the file.
For the error from volcano plot, it seems that there's something wrong in the dea table. Could you double check the dea* tables to make sure that they're alright? If you still get errors, is it OK to share the normalized count tables and dea tables with me? I need the data to debug, because I've never seen that error from my data.


Error deploying rgDelegatedResourceManagement (Code: InvalidRegistrationDefinitionUri)

I'm trying to deploy rgDelegatedResourceManagement using the supplied template and parameter files.
Powershell command:
New-AzResourceGroupDeployment -ResourceGroupName LighthouseTest -TemplateFile .\rgDelegatedResourceManagement.json -TemplateParameterFile .\rgDelegatedResourceManagement.parameters.json
Raw error:
(Subscription id in error is masked by me but matches the target subscription for the deployment)
{
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n  "error": {\r\n    "code": "InvalidRegistrationDefinitionUri",\r\n    "message": "The registration definition request scope '/subscriptions/99999999-9999-9999-9999-999999999999/resourcegroups/LighthouseTest' should be '/subscriptions/<subscriptionId>'."\r\n  }\r\n}"
}
]
}
Parameter file:
(Tenant id and principal id are masked by me below but match the Azure AD tenant id and security group ids for the Azure AD and security group which is requesting delegated permissions)
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "<IP_ADDRESS>",
"parameters": {
"mspName": {
"value": "Breakpoint Technology"
},
"rgName": {
"value": "LighthouseTest"
},
"mspOfferDescription": {
"value": "Breakpoint Technology - AI Analysis"
},
"managedByTenantId": {
"value": "11111111-1111-1111-1111-111111111111"
},
"authorizations": {
"value": [
{
"principalId": "22222222-2222-2222-2222-222222222222",
"principalIdDisplayName": "Lighthouse Test",
"roleDefinitionId": "73c42c96-874c-492b-b04d-ab87d138a893"
}
]
}
}
}
I have confirmed the target subscription has the Microsoft.ManagedServices provider registered.

You are trying to deploy the template to a resource group, but it can only be deployed to a subscription. You must use New-AzDeployment

Good catch.  That resolved it.  Thanks.


Plugin: Add a scaffolding generator for Oni plugin

Was looking through @samvv 's Pencil project, along with the issues: https://github.com/AthenaHQ/Pencil/issues - it's interesting how much overlap there is! 😄
I wanted to port this issue over: athenahq/pencil#3, which is tracking creating a generator to create a plugin. This is really important in order to get people to start building extensions and consuming the API - I like the idea of having a Yeoman generator for creating plugins.

Haha, yeah it really is funny :smile: This project is much is much more advanced, however, that's why I'm making the switch. Getting the community growing is indeed really imortant. It will also require a really good plugin architecture (perhaps for another issue?). I've broken my head in the past how to tackle that problem, which lead to some bizarre constructs such as this one, but till this day I haven't found something of which I can really say "that's it! that'll solve the issue!"


Gate performance.now timestamps behind an experimental flag.

To ensure that using performance.now when reporting timing
values works as expected, gate the behavior behind a global
__spf_experimental_now variable.  Don't use a config as usual
because this value needs to be set before SPF is executed.
Progress on #389.

LGTM just a nit about the function docs.


remove dependency on derefable crate

Crate derefable does not have public repository listed and depends on old versions of syn, quote and proc-macro2 crates.
This change removes the need for derefable crate.
In my other project this change reduced dependency tree by 9 crates and improved compile time by 10%.

I've published 0.2.1 for it.

thank you


updated oee mappings with CDF groups

updating the mappings to CDF groups

messed up the mappings originally with the wrong names


Auto set tad offsets when tad index is specified

Auto set tad offsets when tad index is specified.

Build triggered. sha1 is merged.

Build finished. No test results found.

Refer to this link for build results (access rights to CI server needed):
/job/libnd4j_PR/95/


Make it work without rails

This PR makes the gem work in non-rails project with sensible configuration.
Check updated readme for details.

👍
Tested with rails, still works, all is good 😄

👍


Shaka is pulling second init segment on the second period in Chrome causing audio stutter

Have you read the FAQ and checked for duplicate open issues?
Yes
If the problem is related to FairPlay, have you read the tutorial?

N/A - issue is for Widevine in Chrome
What version of Shaka Player are you using?

4.3.0
Are you using the demo app or your own custom app?
Custom app
If custom app, can you reproduce the issue using our demo app?
Can't test the exact asset with the demo app
What browser and OS are you using?
Chrome on Mac Ventura 13.5.1
For embedded devices (smart TVs, etc.), what model and firmware version are you using?
N/A
What are the manifest and license server URIs?

What configuration are you using?  What is the output of player.getConfiguration()?

{
"drm": {
"retryParameters": {
"maxAttempts": 2,
"baseDelay": 1000,
"backoffFactor": 2,
"fuzzFactor": 0.5,
"timeout": 30000
},
"servers": {
"com.widevine.alpha":
},
"clearKeys": {},
"advanced": {
"com.widevine.alpha": {
"distinctiveIdentifierRequired": false,
"persistentStateRequired": false,
"videoRobustness": "",
"audioRobustness": "",
"serverCertificate": {

},
"individualizationServer": ""
}
},
"delayLicenseRequestUntilPlayed": false,
"logLicenseExchange": false,
"updateExpirationTime": 1
},
"manifest": {
"retryParameters": {
"maxAttempts": 2,
"baseDelay": 1000,
"backoffFactor": 2,
"fuzzFactor": 0.5,
"timeout": 30000
},
"availabilityWindowOverride": null,
"disableAudio": false,
"disableVideo": false,
"disableText": false,
"defaultPresentationDelay": 0,
"dash": {
"clockSyncUri": "",
"ignoreDrmInfo": false,
"xlinkFailGracefully": false,
"ignoreMinBufferTime": false,
"autoCorrectDrift": true,
"initialSegmentLimit": 1000,
"ignoreSuggestedPresentationDelay": false,
"ignoreEmptyAdaptationSet": false,
"ignoreMaxSegmentDuration": false
},
"hls": {
"ignoreTextStreamFailures": false,
"useFullSegmentsForStartTime": true
}
},
"streaming": {
"retryParameters": {
"maxAttempts": 2,
"baseDelay": 1000,
"backoffFactor": 2,
"fuzzFactor": 0.5,
"timeout": 30000
},
"rebufferingGoal": 2,
"bufferingGoal": 1800,
"bufferBehind": 30,
"ignoreTextStreamFailures": false,
"alwaysStreamText": false,
"startAtSegmentBoundary": false,
"smallGapLimit": 0.5,
"jumpLargeGaps": false,
"durationBackoff": 1,
"forceTransmuxTS": false,
"safeSeekOffset": 5,
"stallEnabled": true,
"stallThreshold": 1,
"stallSkip": 0.1,
"useNativeHlsOnSafari": true,
"inaccurateManifestTolerance": 2,
"lowLatencyMode": false,
"autoLowLatencyMode": false,
"forceHTTPS": false,
"preferNativeHls": false
},
"offline": {
"usePersistentLicense": true
},
"abr": {
"enabled": true,
"useNetworkInformation": true,
"defaultBandwidthEstimate": 1000000,
"switchInterval": 8,
"bandwidthUpgradeTarget": 0.85,
"bandwidthDowngradeTarget": 0.95,
"restrictions": {
"minWidth": 0,
"maxWidth": null,
"minHeight": 0,
"maxHeight": null,
"minPixels": 0,
"maxPixels": null,
"minFrameRate": 0,
"maxFrameRate": null,
"minBandwidth": 0,
"maxBandwidth": null
}
},
"preferredAudioLanguage": "",
"preferredTextLanguage": "",
"preferredVariantRole": "",
"preferredTextRole": "",
"preferredAudioChannelCount": 2,
"preferForcedSubs": false,
"restrictions": {
"minWidth": 0,
"maxWidth": null,
"minHeight": 0,
"maxHeight": null,
"minPixels": 0,
"maxPixels": null,
"minFrameRate": 0,
"maxFrameRate": null,
"minBandwidth": 0,
"maxBandwidth": null
},
"playRangeStart": 0,
"playRangeEnd": null
}
What did you do?

Play multi-period DASH manifest content on Chrome. Shaka player is fetching the init segment again and reloading the playback pipeline when period continuity is signaled in the manifest.
What did you expect to happen?
Shaka player does not fetch the init segment again and reload the playback pipeline on the second period when period continuity is signaled in the manifest.
What actually happened?

Audio stutters between first and second period boundary

@nkam12 Please provide a manifest. If not, we can't help you. Thanks!


dependabot

Please read contributing guidelines and remove this line.

#5547  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @Maddiaa0 and the rest of your teammates on  Graphite

Benchmark results
Metrics with a significant change:

note_successful_decrypting_time_in_ms (8): 241 (+16%)
note_trial_decrypting_time_in_ms (8): 47.2 (+288%)
note_trial_decrypting_time_in_ms (32): 94.8 (+86%)

Detailed results
All benchmarks are run on txs on the Benchmarking contract on the repository. Each tx consists of a batch call  to create_note and increment_balance, which guarantees that each tx has a private call, a nested private call, a public call, and a nested public call, as well as an emitted private note, an unencrypted log, and public storage read and write.
This benchmark source data is available in JSON format on S3 here.
Values are compared against data from master at commit 0249737e and shown if the difference exceeds 1%.
L2 block published to L1
Each column represents the number of txs on an L2 block published to L1.

Metric
8 txs
32 txs
64 txs

l1_rollup_calldata_size_in_bytes
676
676
676

l1_rollup_calldata_gas
6,424
6,424
6,400

l1_rollup_execution_gas
585,757
585,757
585,733

l2_block_processing_time_in_ms
1,315 (-1%)
4,772
9,179 (+1%)

note_successful_decrypting_time_in_ms
:warning: 241 (+16%)
584 (+8%)
1,001 (+2%)

note_trial_decrypting_time_in_ms
:warning: 47.2 (+288%)
:warning: 94.8 (+86%)
120 (+20%)

l2_block_building_time_in_ms
11,485
42,305 (+1%)
83,891 (+2%)

l2_block_rollup_simulation_time_in_ms
7,434
26,305 (+1%)
51,398

l2_block_public_tx_process_time_in_ms
4,032 (+1%)
15,948 (+3%)
32,386 (+4%)

L2 chain processing
Each column represents the number of blocks on the L2 chain where each block has 16 txs.

Metric
5 blocks
10 blocks

node_history_sync_time_in_ms
14,086 (-2%)
26,536 (-2%)

note_history_successful_decrypting_time_in_ms
1,269 (-3%)
2,256 (-8%)

note_history_trial_decrypting_time_in_ms
60.2 (-41%)
79.0 (-31%)

node_database_size_in_bytes
18,591,824
34,934,864

pxe_database_size_in_bytes
29,859
59,414

Circuits stats
Stats on running time and I/O sizes collected for every circuit run across all benchmarks.

Circuit
circuit_simulation_time_in_ms
circuit_input_size_in_bytes
circuit_output_size_in_bytes

private-kernel-init
180 (-2%)
44,379
28,214

private-kernel-ordering
168 (-1%)
52,880
14,296

base-parity
3,427 (-1%)
128
311

root-parity
1,268 (-3%)
1,244
311

base-rollup
14,823 (-1%)
165,760
861

root-rollup
49.2 (-1%)
4,359
725

private-kernel-inner
224 (-1%)
73,795
28,214

public-kernel-app-logic
96.4 (+1%)
35,252
28,217

public-kernel-tail
138 (+1%)
40,928
28,217

merge-rollup
9.30 (+1%)
2,568
861

public-kernel-teardown
93.4
35,252
28,217

public-kernel-setup
92.3 (+1%)
35,252
28,217

Tree insertion stats
The duration to insert a fixed batch of leaves into each tree type.

Metric
1 leaves
16 leaves
64 leaves
128 leaves
512 leaves
1024 leaves
2048 leaves
4096 leaves
32 leaves

batch_insert_into_append_only_tree_16_depth_ms
9.90 (-1%)
15.9 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_count
16.8
31.6
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_16_depth_hash_ms
0.576 (-1%)
0.489 (-1%)
N/A
N/A
N/A
N/A
N/A
N/A
N/A

batch_insert_into_append_only_tree_32_depth_ms
N/A
N/A
46.8 (-2%)
71.4 (-1%)
233
432 (-3%)
875
1,723
N/A

batch_insert_into_append_only_tree_32_depth_hash_count
N/A
N/A
96.0
159
543
1,055
2,079
4,127
N/A

batch_insert_into_append_only_tree_32_depth_hash_ms
N/A
N/A
0.480 (-2%)
0.439 (-1%)
0.423
0.403 (-3%)
0.415
0.413
N/A

batch_insert_into_indexed_tree_20_depth_ms
N/A
N/A
53.5 (-2%)
106 (-1%)
344 (+2%)
641 (-3%)
1,316 (+1%)
2,610
N/A

batch_insert_into_indexed_tree_20_depth_hash_count
N/A
N/A
105
207
691
1,363
2,707
5,395
N/A

batch_insert_into_indexed_tree_20_depth_hash_ms
N/A
N/A
0.471 (-2%)
0.476 (-2%)
0.466 (+1%)
0.441 (-3%)
0.455
0.454
N/A

batch_insert_into_indexed_tree_40_depth_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
60.1 (-2%)

batch_insert_into_indexed_tree_40_depth_hash_count
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
109

batch_insert_into_indexed_tree_40_depth_hash_ms
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
0.525 (-2%)

Miscellaneous
Transaction sizes based on how many contract classes are registered in the tx.

Metric
0 registered classes
1 registered classes

tx_size_in_bytes
14,968
495,320

Transaction size based on fee payment method

Metric
none fee payment method
native fee payment method
fpc_public fee payment method
fpc_private fee payment method

tx_with_fee_size_in_bytes
773
905
1,161
1,377

Transaction processing duration by data writes.

Metric
0 new note hashes
1 new note hashes
2 new note hashes

tx_pxe_processing_time_ms
1,801
1,109 (-1%)
5,513

Metric
0 public data writes
1 public data writes
2 public data writes
3 public data writes
4 public data writes
5 public data writes
8 public data writes

tx_sequencer_processing_time_ms
10.9 (-5%)
490 (+1%)
380 (-1%)
917
562 (+1%)
1,504
549 (-1%)


HBASE-28651 Reformat the javadoc for CellChunkMap

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
0m 25s
Docker mode activated.


Payment page requires rekeying  of address

Users are having to rekey their address twice:

this is a duplicate of #235


[SPARK-32552][SQL][DOCS]Complete the documentation for Table-valued Function

What changes were proposed in this pull request?
There are two types of TVF. We only documented one type. Adding the doc for the 2nd type.
Why are the changes needed?
complete Table-valued Function doc
Does this PR introduce any user-facing change?

How was this patch tested?
Manually build and check

Test build #127069 has started for PR 29355 at commit fda6ecf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31691/
Test PASSed.

cc @maropu

Test build #127069 has finished for PR 29355 at commit fda6ecf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127069/
Test PASSed.

Thanks! I'll check it in hours.

Ah, could you file a new JIRA ticket just in case?

https://github.com/apache/spark/pull/28185/files#r464239662

Here is the link for HIVE Built-in Table-Generating Functions (UDTF). Should we also include json_tuple and parse_url?

Here is the link for HIVE Built-in Table-Generating Functions (UDTF). Should we also include json_tuple and parse_url?

yea, I think we should.

@maropu Thanks for your review! I have addressed the comments. Could you please take another look?

Test build #127161 has started for PR 29355 at commit 292eb02.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/31783/
Test PASSed.

Test build #127161 has finished for PR 29355 at commit 292eb02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127161/
Test PASSed.

@maropu Thanks for your review! I have addressed the comments. Could you please take another look?

Thanks for the update, @huaxingao ! Yea, I'l check it later (in hours).

retest this please

Test build #127822 has started for PR 29355 at commit 292eb02.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/32447/
Test FAILed.

Thanks, @huaxingao ! Merged to master/branch-3.0. cc: @gatorsmile

Test build #127822 has finished for PR 29355 at commit 292eb02.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127822/
Test PASSed.

Thanks a lot! @maropu