Fhicl files for 50% DS field and positive muon beam

Hi Everyone,
I've added epilog files in JobConfig/beam and JobConfig/pileup for:

negative muon beam and 50% DS field,
positive muon beam and 100% DS field, and
positive muon beam and 50% DS field

As well as versions of all the main fcl files in both of those folders (e.g. POT_mubeam.fcl, EleBeamResampler_DS50.fcl, and NeutralsResampler_mubeam_DS50.fcl.
I held off from going into the primary directory for the moment because I realised there would be even more copy and pasting. Do we want four versions of every file, or do we want to do something clever with the scripts that run the generate_fcl command (e.g. a command-line option that will #include the correct epilog file)? The former is much easier for people to find and run what they need for local tests, but I think the latter will be easier to maintain.
Cheers,
Andy

Hi @AndrewEdmonds11,
You have proposed changes to files in these packages:

JobConfig

which require these tests: build.
@Mu2e/fnalbuild-users, @Mu2e/write have access to CI actions on main.
:hourglass: The following tests have been triggered for ae20f704d4abf0adf74b3dc2072f263cf10c6914: build  (Build queue has 2 jobs)
About FNALbuild. Code review on Mu2e/Offline.

:sunny: The tests passed at ae20f704d4abf0adf74b3dc2072f263cf10c6914.

Test
Result
Details

merge
:white_check_mark:
Merged ae20f704d4abf0adf74b3dc2072f263cf10c6914 at 294d2dff9b5770b5a5830fc680a574c98c4e54e6

build (prof)
:white_check_mark:
Log file. Build time: 11 min 56 sec

ceSimReco
:white_check_mark:
Log file.

g4test_03MT
:white_check_mark:
Log file.

transportOnly
:white_check_mark:
Log file.

POT
:white_check_mark:
Log file.

g4study
:white_check_mark:
Log file.

cosmicSimReco
:white_check_mark:
Log file.

cosmicOffSpill
:white_check_mark:
Log file.

ceSteps
:white_check_mark:
Log file.

ceDigi
:white_check_mark:
Log file.

muDauSteps
:white_check_mark:
Log file.

ceMix
:white_check_mark:
Log file.

rootOverlaps
:white_check_mark:
Log file.

g4surfaceCheck
:white_check_mark:
Log file.

FIXME, TODO
:white_check_mark:
TODO (0) FIXME (0) in 0 files

clang-tidy
:white_check_mark:
0 errors 0 warnings

N.B. These results were obtained from a build of this Pull Request at ae20f704d4abf0adf74b3dc2072f263cf10c6914 after being merged into the base branch at 294d2dff9b5770b5a5830fc680a574c98c4e54e6.
For more information, please check the job page here.
Build artifacts are deleted after 5 days. If this is not desired, select Keep this build forever on the job page.

Hi Andy,
I suggest to implement reduced field options as epilogue overrides, based
in JobConfig/common.  They can then be added when we build the final
scripts for grid running, the same way we append the different beam
intensity models in mixing, see JobConfig/Scripts/gen_Mix.sh
Cheers, Dave
On Fri, Apr 8, 2022 at 06:57 Andrew Edmonds @.***>
wrote:

Hi Everyone,
I've added epilog files in JobConfig/beam and JobConfig/pileup for:

negative muon beam and 50% DS field,
positive muon beam and 100% DS field, and
positive muon beam and 50% DS field

As well as versions of all the main fcl files in both of those folders
(e.g. POT_mubeam.fcl, EleBeamResampler_DS50.fcl, and
NeutralsResampler_mubeam_DS50.fcl.
I held off from going into the primary directory for the moment because I
realised there would be even more copy and pasting. Do we want four
versions of every file, or do we want to do something clever with the
scripts that run the generate_fcl command (e.g. a command-line option
that will #include the correct epilog file)? The former is much easier
for people to find and run what they need for local tests, but I think the
latter will be easier to maintain.
Cheers,
Andy
You can view, comment on, or merge this pull request online at:
https://github.com/Mu2e/Production/pull/166
Commit Summary

a4c47a3
https://github.com/Mu2e/Production/pull/166/commits/a4c47a354259cd631032dadf2329d9ccffdb736f
Adding fcl files for POT with muplus beam and 50% DS field
2425206
https://github.com/Mu2e/Production/pull/166/commits/2425206ee950427dd141064d80827a625da578fb
Add fcl files for DS50 and muplus_DS50 in the pileup folder
45a0f47
https://github.com/Mu2e/Production/pull/166/commits/45a0f47310dce7cedfe0cb035623c36423a86e33
Update POT fcl files to use epilogs
b13069f
https://github.com/Mu2e/Production/pull/166/commits/b13069fbb8c62a826be08235fc023cf8a2b0573b
Add positive muon and 50% DS field POT fcls for the infinte pion case
ae20f70
https://github.com/Mu2e/Production/pull/166/commits/ae20f704d4abf0adf74b3dc2072f263cf10c6914
Add missing muplus and 100% DS field fcl files to pileup folder

File Changes
(27 files https://github.com/Mu2e/Production/pull/166/files)

A JobConfig/beam/POT_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-bc13856907b711c2373cf7ce8da014ce9922aca7e1c804ec59e9d2ac0c1d3356
(2)
A JobConfig/beam/POT_infinitepion_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-767e1947ac68455ec5f41ee51bb900c419bc60408f070403f6b1ac0ec5e537e3
(2)
A JobConfig/beam/POT_infinitepion_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-c7d90b4868a32d9c90097d569e6d111bcd676b674657230213917508cf0868a4
(2)
A JobConfig/beam/POT_infinitepion_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-6cd9e3719ce9d3e94c2573a341fd0e0f34a613780f89177c0d816683904bdb83
(2)
A JobConfig/beam/POT_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-8d66f8acbaa15141b10243a785f9bf35fefb18b7920b484213d5b13f6fb1d345
(2)
A JobConfig/beam/POT_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-1ca1790a95463b121c8fa6d3f6236da3240491f16e8a621bd152cfb8abbcd400
(2)
A JobConfig/beam/epilog_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-e980fb108be114e33487f9422771ca89e23f1b8de6465db5f1d4f564b303810e
(4)
A JobConfig/beam/epilog_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-e54b13b8006508701eb47ef034b4a25898d19c16112d9a8cf47f8998eb2f2733
(4)
A JobConfig/beam/epilog_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-73da3e60a710972d93425c83b414c2c4bda3023ee9b781ad427c4d3e80dd482c
(5)
A JobConfig/pileup/EleBeamResampler_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-f2112cd4606577ba11e44804873fc94d72702583cd81266b42776c7d05a22be0
(2)
A JobConfig/pileup/EleBeamResampler_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-7052900f54bd5ab3da600c0d9ceb43fc4a2a166d101b5444047b1e41752938ab
(2)
A JobConfig/pileup/EleBeamResampler_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-8a1ffa3a33167c56afd0f06a7d9e0b7d09dd9673926a2ba13fed9617fdb3d801
(2)
A JobConfig/pileup/MuBeamResampler_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-09cb503544c11732fb71961f957791a6fbc2f27aff0a17918f3269dba51469a3
(2)
A JobConfig/pileup/MuBeamResampler_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-614a8066ab637738924422d3f08dd09208d88bc938e163c96207d0b0bded8b1d
(2)
A JobConfig/pileup/MuBeamResampler_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-a59be611e6014b07f82e20244f3469e3a519a833ae3b75374b671133c6c96687
(2)
A JobConfig/pileup/MuStopPileup_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-1eba7287eb9b1f6ff463b1788176bff9287e5ea0038cec1371ba7dae76d0f092
(2)
A JobConfig/pileup/MuStopPileup_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-3be9ba9184c2abd37b904507ee5eb1e099c6067ac2817dc70172a09295a66e8c
(2)
A JobConfig/pileup/MuStopPileup_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-acb1706323f9a322aea2f632913e4c771a2cf62bdc1a6eec7bb5bb72a06c2e5d
(2)
A JobConfig/pileup/NeutralsResampler_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-b3563fa5e176958671cabbf4445959ee34e5f51c2152a6566a28c3d12c22b64c
(2)
A JobConfig/pileup/NeutralsResampler_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-9489ccdcbbd61cba61e2504949e0a186ad0e368542c8ae6c2b7deaf40e43d6d8
(2)
A JobConfig/pileup/NeutralsResampler_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-52460eb2a3c1bff80a8770ab736aa93333772cdc59ed71a91cd701190906c3eb
(2)
A JobConfig/pileup/PiBeamResampler_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-9c69181a278f2b69df65a8a93b7ee517de569e2e887d16d65556fc9857683922
(2)
A JobConfig/pileup/PiBeamResampler_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-ed215e5153a5e685b47280d6318126df1cc7607bbe194aa8b43c3f3c09531f1a
(2)
A JobConfig/pileup/PiBeamResampler_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-7625d9a63e56e3d076ce5bab6888070ea67118f54c9ffdc7fad6c60e6273030c
(2)
A JobConfig/pileup/epilog_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-2b5be9d69276585b9e3d4414ece707bd4644cde7ab651d38c372af1782e529dc
(4)
A JobConfig/pileup/epilog_muplus.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-b5394247e0159574bdb5500cd8b3c49ed1e6f57318c74e7fe798e06ab62eb4e8
(4)
A JobConfig/pileup/epilog_muplus_DS50.fcl
https://github.com/Mu2e/Production/pull/166/files#diff-9d72f8c180906ac646a9e3bc3afbbef3c00adb6fcbbbf7ad684eff2c29c284b9
(5)

Patch Links:

https://github.com/Mu2e/Production/pull/166.patch
https://github.com/Mu2e/Production/pull/166.diff

—
Reply to this email directly, view it on GitHub
https://github.com/Mu2e/Production/pull/166, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABAH575ENANAO4LZCVVUFBLVEA3NBANCNFSM5S4UN4FA
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

--
David Brown            @.***
Office Phone<PHONE_NUMBER>
Lawrence Berkeley National Lab
M/S 50R5008 (50-6026C)  Berkeley, CA 94720

Thanks, Dave. I think I have it all set up as you suggested. This PR now only has epilog files in JobConfig/common and no other changes.


'checklinks' error: undefined method `split' for nil:NilClass

$ brew cask checklinks
Error: undefined method `split' for nil:NilClass
Please report this bug:
    https://github.com/caskroom/homebrew-cask/issues
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/underscore_supporting_uri.rb:20:in `simple_parse'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/underscore_supporting_uri.rb:8:in `rescue in parse'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/underscore_supporting_uri.rb:5:in `parse'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/url.rb:12:in `initialize'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/dsl.rb:58:in `new'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/dsl.rb:58:in `url'
<main>:4:in `<class:Test>'
<main>:in `<main>'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/source/path_base.rb:55:in `eval'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/source/path_base.rb:55:in `rescue in load'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/source/path_base.rb:52:in `load'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/source/tapped.rb:28:in `load'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask.rb:127:in `load'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/scopes.rb:8:in `block in all'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/scopes.rb:8:in `map'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/scopes.rb:8:in `all'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/cli/checklinks.rb:3:in `run'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/cli.rb:80:in `run_command'
/usr/local/Cellar/brew-cask/0.39.3/rubylib/cask/cli.rb:120:in `process'
/usr/local/bin/brew-cask.rb:42:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
/usr/local/Library/brew.rb:59:in `require?'
/usr/local/Library/brew.rb:137:in `<main>'

$ brew cask doctor
==> OS X Version:
10.9.4
==> Hardware Architecture:
intel-64
==> Ruby Version:
2.0.0-p451
==> Ruby Path:
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby
==> Homebrew Version:
0.9.5
==> Homebrew Executable Path:
/usr/local/bin/brew
==> Homebrew Cellar Path:
/usr/local/Cellar
==> Homebrew Repository Path:
/usr/local
==> Homebrew Origin:
https://github.com/Homebrew/homebrew
==> Homebrew-cask Version:
0.39.3
==> Homebrew-cask Install Location:
/usr/local/Cellar/brew-cask/0.39.3
==> Homebrew-cask Cached Downloads:
/Library/Caches/Homebrew
/Library/Caches/Homebrew/Casks
10 files (warning: run "brew cask cleanup")
971.30 megs (warning: run "brew cask cleanup")
==> Homebrew-cask Default Tap Path:
/usr/local/Library/Taps/caskroom/homebrew-cask
==> Homebrew-cask Alternate Cask Taps:
/usr/local/Library/Taps/caskroom/homebrew-fonts
/usr/local/Library/Taps/caskroom/homebrew-versions
==> Homebrew-cask Default Tap Cask Count:
1842
==> Contents of $LOAD_PATH:
/usr/local/Cellar/brew-cask/0.39.3/rubylib
/usr/local/Library/Homebrew
/Library/Ruby/Site/2.0.0
/Library/Ruby/Site/2.0.0/x86_64-darwin13
/Library/Ruby/Site/2.0.0/universal-darwin13
/Library/Ruby/Site
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/vendor_ruby/2.0.0
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/vendor_ruby/2.0.0/x86_64-darwin13
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/vendor_ruby/2.0.0/universal-darwin13
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/vendor_ruby
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/x86_64-darwin13
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/universal-darwin13
==> Contents of $RUBYLIB Environment Variable:
<NONE>
==> Contents of $RUBYOPT Environment Variable:
<NONE>
==> Contents of $RUBYPATH Environment Variable:
<NONE>
==> Contents of $RBENV_VERSION Environment Variable:
<NONE>
==> Contents of $CHRUBY_VERSION Environment Variable:
<NONE>
==> Contents of $GEM_HOME Environment Variable:
<NONE>
==> Contents of $GEM_PATH Environment Variable:
<NONE>
==> Contents of $BUNDLE_PATH Environment Variable:
<NONE>
==> Contents of $PATH Environment Variable:
PATH="/usr/local/sbin:/Users/name/.rvm/gems/ruby-2.0.0-p481/bin:/Users/name/.rvm/gems/ruby-2.0.0-p481@global/bin:/Users/name/.rvm/rubies/ruby-2.0.0-p481/bin:/Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Users/name/.rvm/bin:/usr/local/Library/Contributions/cmd"
==> Contents of $SHELL Environment Variable:
SHELL="/bin/bash"
==> Contents of Locale Environment Variables:
LC_CTYPE="UTF-8"
==> Running As Privileged User:
No

checklinks is removed on 0.53.0
$ brew cask checklinks
brew-cask provides a friendly homebrew-style CLI workflow for the
administration of Mac applications distributed as binaries.

!!
!! no command verb: checklinks
!!

Commands:

    audit      verifies installability of Casks
    cat        dump raw source of the given Cask to the standard output
    cleanup    cleans up cached downloads and tracker symlinks
    create     creates the given Cask and opens it in an editor
    doctor     checks for configuration issues
    edit       edits the given Cask
    fetch      downloads Cask resources to local cache
    home       opens the homepage of the given Cask
    info       displays information about the given Cask
    install    installs the given Cask
    list       with no args, lists installed Casks; given installed Casks, lists staged files
    search     searches all known Casks
    uninstall  uninstalls the given Cask
    update     a synonym for 'brew update'
    zap        zaps all files associated with the given Cask

See also "man brew-cask"


add data corrections for "sizable minority" utilities

PR Overview
Working on  #3064
i need some naming help in here in general but for these things specifically:

what to call this phenomenon? I've been going with sizable_minority and off_by as short hands to indicate that a sizable minority of utilities are reporting some calculable facts off-by another fact. This both includes utilities including child facts into calcs when that is not standard, as well as utilities excluding child facts.
OffByFactoid: right now we can enumerate facts that are off by a child fact with this class.
The correction factoid name: x.xbrl_factoid + "_off_by_" + x.xbrl_factoid_off_by + "_correction"... this is loooong. but informative. For reference, the standard calc corrections are just x.xbrl_factoid + "_correction". In the context of the explode_calcs I could image it just being "off_by_" + x.xbrl_factoid_off_by + "_correction" because the xbrl_factoid_parent is also a column, but in the exploded data that context isn't included.

outcomes:

the three off-by facts to fix i've added so far dropped the non-null error rate down by more than 10% but it's still high:

2023-11-29 10:22:54 [    INFO] catalystcoop.pudl.transform.ferc1:1487 balance_sheet_assets_ferc1: Correcting 44.05% of all non-null reported values (9759/22152) out of a total of 936665 original records.

before:
2023-11-29 15:42:28 [    INFO] catalystcoop.pudl.transform.ferc1:1486 balance_sheet_liabilities_ferc1: Correcting 6.91% of all non-null reported values (226/3271) out of a total of 323961 original records.
2023-11-29 15:42:52 [    INFO] catalystcoop.pudl.transform.ferc1:1486 income_statement_ferc1: Correcting 17.80% of all non-null reported values (5235/29413) out of a total of 1207814 original records.
2023-11-29 15:47:29 [    INFO] catalystcoop.pudl.transform.ferc1:1486 balance_sheet_assets_ferc1: Correcting 54.94% of all non-null reported values (12171/22152) out of a total of 936669 original records.

PR Checklist

[x] Merge the most recent version of the branch you are merging into (probably dev).
[ ] All CI checks are passing. Run tests locally to debug failures
[ ] Make sure you've included good docstrings.
[ ] For major data coverage & analysis changes, run data validation tests
[ ] Include unit tests for new functions and classes.
[ ] Defensive data quality/sanity checks in analyses & data processing functions.
[ ] Update the release notes and reference reference the PR and related issues.
[ ] Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

Blocking: Confused about the presence of environment changes in this PR.
I was also confused about this! I asked @zaneselvans  and it seems like small changes to the lock files with minor version updates should be expected in general. I believe this is because we have so many dependencies that it is likely that from day to day we'll see small version updates.

Non-blocking: We calculate, add correction factoids, then calculate everything again. I am not sure whether it'd just be simpler to calculate once, add the correction factoids and change the corrected abs_diffs to 0 before proceeding to reconciling calcs?

I was wondering about this. I think I could break up reconcile_intertable_calculations into two steps so boom could include something like:
    exploded = (
            self.initial_explosion_concatenation(tables_to_explode)
            .pipe(self.calculate_values_from_components)
            .pipe(self.add_sizable_minority_corrections) # which would have to correct the abs_diffs...
            .pipe(self.reconcile_intertable_calculations)
            .pipe(self.calculation_forest.leafy_data, value_col=self.value_col)
        )

I will try this. I'm not sure this will be simpler even though it would mean we would run pudl.transform.ferc1.calculate_values_from_components only once.

okay @e-belfer I was able to address the non-blocking concern so I tink its ready for a re-review.


Use CircleCI deployment

instead of running at the end of the arch tests

Removing the v1.0.0 milestone as it is just a technicality with no influence on the project functionality or API.

Closing as we use more CircleCI-ish way for our PyPI deploy now.


Use precommit githook to prevent addition of unlinted and untested code

Best bet here would be to use the husky package on npm. Its by far the most widely adopted.
Once this is in place drastic changes will become more difficult to implement, so it'd be best if this be held off till test coverage is improved. Plus this is blocked by #14 and #15.


seed staff user in review apps

Looks great, useful stuff

Doesn't actually work. not sure why. also since i don't have access i can't pinpoint why

not needed for now


How to handle hot and cold data storage

Describe your question
How does openGemini handle hierarchical storage of hot and cold data? Is this an implemented feature? I didn't find any specific introduction in the documentation. Can you explain how this is achieved? Thank you so much

cc @xiangyu5632

I saw someone doing a dual-write solution at the application layer, one for the time series database and one for Hadoop.

Thank you for your answer. I prefer to implement it in double writing at the application layer. @xiangyu5632


Allow passing in traitlets via commandline

Without this, you always needed a repo2docker_config.py
file to configure anything. This PR makes r2d match the
behavior of most traitlets based applications (like jupyter_server,
jupyterhub, nbconvert, etc)
Fixes https://github.com/jupyterhub/repo2docker/issues/1112

Do you know how to get the --help-all argument working? E.g. for jupyterhub
$ jupyterhub --help
...
Options
=======
The options below are convenience aliases to configurable class-options,
as listed in the "Equivalent to" description-line of the aliases.
To see all configurable class-options for some <cmd>, use:
    <cmd> --help-all

...

$ jupyterhub --help-all
...
Class options
=============
The command-line option below sets the respective configurable class-parameter:
    --Class.parameter=value
This line is evaluated in Python, so simple expressions are allowed.
For instance, to set `C.a=[0,1,2]`, you may type this:
    --C.a='range(3)'

Application(SingletonConfigurable) options
------------------------------------------
--Application.log_datefmt=<Unicode>
    The date format used by logging formatters for %(asctime)s
    Default: '%Y-%m-%d %H:%M:%S'

...

@manics so we're mixing argparse and traitlets here, which is causing some of these issues. Ideally we would just be using traitlets and this will go away... I'm not entirely sure how to get --help-all working, other than to try get rid of argparse.

I added --help-all. Application().print_help() is the method to call (classes=True is --help-all).
It does get a little funky, since there are two full argparse CLI instances running now (the one generated by traitlets, and our handcrafted one). I think if we want to have full traitlets CLI parsing, perhaps we should use traitlets CLI parsing. I understand not wanting to do that (it's simpler and nicer to write your own parser), but if we want the --Class.trait=value magic, it might be better to go all-in. We ought to get things like --show-config and friends for free, too.

CI failure is unrelated
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-notebook", line 5, in <module>
    from notebook.notebookapp import main
  File "/usr/local/lib/python3.5/site-packages/notebook/notebookapp.py", line 134, in <module>
    from .terminal import TerminalManager
  File "/usr/local/lib/python3.5/site-packages/notebook/terminal/__init__.py", line 4, in <module>
    import terminado
  File "/usr/local/lib/python3.5/site-packages/terminado/__init__.py", line 7, in <module>
    from .websocket import TermSocket
  File "/usr/local/lib/python3.5/site-packages/terminado/websocket.py", line 87
    self.log_terminal_output(f'STDOUT: {content[1]}')

Also occurred on main this morning 😞 https://github.com/jupyterhub/repo2docker/runs/4963525952?check_suite_focus=true

@minrk agree going all inn is probably a good idea. Do you want me to do that to move this PR forward? Or can / should it be deferred?

Or can / should it be deferred?

It's fine to defer it.

CI failure is unrelated

The test failure is a metadata bug in terminado 0.13. Should be fixed by https://github.com/jupyter/terminado/pull/140 once 0.13.0 is yanked.

Green! :rocket:
Thanks all!

Follow-up issue here: https://github.com/jupyterhub/repo2docker/issues/1129


Feature/Add action file for auto docker image build

Added script and github actions to auto build the dev and release docker images as well as send them to jfrog

Wrong destination


Fix persisting Witness state for iOS

Overview
iOS filesystem service crashes when it attempts to save state. We need to solve two problems:
Prevent application crash from platform-specific services
Platform-specific services have higher than usual chance to crash due to differences between platforms, permissions and such. Thus those services should always be treated in a way that prevents application crashes.
Fix file system storage on iOS

Implemented


⚠ Noco has degraded performance

In 3cf1798, Noco (https://nocodb.sps-platforma.ru) experienced degraded performance:

HTTP code: 200
Response time: 6528 ms

Resolved: Noco performance has improved in 586a1ba after 9 minutes.


notify if openvr_api.dll's checksum doesn't match

this would be useful when they're overwritten by updates

This is addressed in this release
https://github.com/tappi287/openvr_fsr_app/releases/tag/0.6.3


Preview debug

closes #192
See #192 for some background on problem, and how I narrowed it down to a preview bug and not a CDN + caching one.
There are 3 issues resolved here:
1: Currently, if you save a theme option headless variable in production, it is ignored (it's only checked on staging). But sometimes we want to test a preview url in prod, like this instance here for debugging.
2: Currently, on an already published page (e.g. post_id = 9, if you make a change and hit preview, you'll be redirected to the revision id, e.g. /96. This will then grab the correction preview revision data. However when you click edit, it takes you to https://bubsnext.wpengine.com/wp-admin/post.php?action=edit&post=96, which is an error as it's not a post id. The URL we want to edit instead is:
https://bubsnext.wpengine.com/wp-admin/revision.php?revision=96
3: If the page status is draft, and you make a revision then go to preview, the correct behavior is redirect the to go to /{revisionID}, e.g. /95. Currently however it sends you to /3, which loads the orginally saved draft, but doesn't grab your changes.
To recreate any of the above, set the theme option for preview URL to https://bubs.patronage.org. To verify fixes, set to https://bubs-next-git-preview-debug-patronage.vercel.app

One question I had for @kylehotchkiss @chrisherold -- I setup WORDPRESS_REVISION_URL to match the existing WORDPRESS_EDIT_URL, but I can't find any instance of us actually using that. Can we remove the reference to that assumed ENV, and just rely on import { WORDPRESS_URL } from 'lib/constants';?

One question I had for @kylehotchkiss @chrisherold -- I setup WORDPRESS_REVISION_URL to match the existing WORDPRESS_EDIT_URL, but I can't find any instance of us actually using that. Can we remove the reference to that assumed ENV, and just rely on import { WORDPRESS_URL } from 'lib/constants';?

I'm a fan of using WORDPRESS_URL and building on top of it around the next app. it looks like revision url and edit url are going to be consistent throughout the codebase and our projects so no need to overwrite them with env vars.


Complete a Baked AMI image

work with Mark to get a completed and baked AMI image

[x] Pipeline for base image

should be a change to do that today since we completed and install yesterday with Mark B

base pipeline will be completed today
push GLUU and Database to next sprint

AWS migration project cancelled.


Update clap usage for most recent HEAD version

These are the changes I had to make to get this to compile with the most recent
version of zig-clap.

:pray:


Filter with check against list of tuples

We are missing support for the following case.
Example syntax:

case class Person(id: Int, name: String, age: Int)

val tuples: Seq[(String, Int)] = Seq(("Foo", 1), ("Bar", 2))

ctx.run(quote[Person].filter(t => lift(tuples).contains((t.name, t.age))))

that for Postgres compiles to:
SELECT id, name, age
FROM person t
WHERE (t.name, t.age) in (('Foo', 1), ('Bar', 2))

and for other databases that do not support such syntax to:
SELECT id, name, age
FROM person t
WHERE t.name = 'Foo' AND t.age = 1 OR t.name = 'Bar' AND t.age = 2

@getquill/maintainers

Seems it's similar to https://github.com/getquill/quill/issues/615.

workaround: https://scastie.scala-lang.org/qpdPCa2QQoODJ9Q3qfcquw

Looks like the workaround on Scastie is gone. Would you mind reposting it?

Could you share a workaround again?

@jcoughlin9 A version that survived in my code: https://gist.github.com/sgrankin/1862c194c6544be10dceb4b19a1259f3
Also note that on mysql 5.6 that construction may result in table scans; I've been trying out a different construct https://gist.github.com/sgrankin/b6fe659aacb93e57b4483dbfc1377e49 but it has has a bad quill runtime for non-trivial input sizes. (Patches forthcoming once I had time to fully test them).

A tail recursive version of liftQuery.
https://gist.github.com/sujeet100/74346963bb1281619a08f9960a46e24c

I just hit this in a work project, hope someone can figure out a solution :)

I know it was long time ago, but have you considered to address this issue in the future?

@guizmaii I know it was long time ago, but have you considered to address this issue in the future?


I want to create a custome filters like "friends", "office colleagues", "leads", etc. How can i create one

Filters are not working.
Is there any way to filter chats based.
Describe the solution you'd like
I can create a custome filter and add contact/chats in it. Such that it can organise my contacts. There can be some method or way or function through which I can create new filter and chatlist
Describe alternatives you've considered
Till now, we are using __x_shouldAppearInList, shouldAppearInList method.

Please, test the PR https://github.com/wppconnect-team/wa-js/pull/2498

Already merge PR, you can use WPP.chat.setChatList function


Inconsistent type definitions between retrieving account and silent request [TS]

Core Library
MSAL.js v2 (@azure/msal-browser)
Core Library Version
2.14.2
Wrapper Library
MSAL React (@azure/msal-react)
Wrapper Library Version
1.0.0
Description
Using React Typescript, when retrieving the active account from the MSAL instance, the type definition is AccountInfo | Null.
While e.g. the SilentRequest account property requires a type of AccountInfo | Undefined, causing the need of something like this:

This currently forces me to re-map the account object when passing it in for silent requests, which seems like a slightly unnecessary step.
It would be beneficial if we could have consistent definitions, that don't require any extra steps of converting between null and undefined.
Error Message
No response
Msal Logs
No response
MSAL Configuration
auth: {
        clientId: '',
        redirectUri: '',
        postLogoutRedirectUri: '',
        knownAuthorities: ['']
    },

Relevant Code Snippets
getAccessToken(context: IMSalContext) {
  const {instance} = context;
  const account = instance.getActiveAccount();
  instance.acquireTokenSilent({
            ...loginRequest,
            account: account
        })
 ...
}

Reproduction Steps

Get the active account from your MSAL context
Attempt to pass the account as a property to any SilentRequest

Expected Behavior
Expecting consistent type definitions across different functions
Identity Provider
Azure B2C Basic Policy
Browsers Affected (Select all that apply)
Chrome, Firefox, Edge, Safari, Internet Explorer, Other
Regression
No response
Source
External (Customer)

the SilentRequest account property requires a type of AccountInfo | Undefined

Silent calls have AccountInfo as a mandated parameter. I believe this is converted to AccountInfo/Undefined post compilation?

Using React Typescript, when retrieving the active account from the MSAL instance, the type definition is AccountInfo | Null.

getActiveAccount() can return a null, since there is a possibility of no accounts in the cache/memory.
I can check if there is a better way to avoid this.


RFP - Helm

Homepage: https://helm.sh/
Description: The package manager for Kubernetes

Package exists and is actively maintained:
https://chocolatey.org/packages/kubernetes-helm

@joachimschmidt557 thanks for checking this!

looks like it didn't when the request was created though, but nonetheless thanks for checking up on this.


[Enum Request] Table 99000770 Manufacturing Comment Line: Table Name Type Option to Enum

Can you please convert the Field 1 "Table Name" from a Option to an extensible Enum?

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: .


Fix avatar blurriness in profile screen

Description
Updated the avatar URL used in the profile screen to request a higher-resolution image to reduce blurriness.
Fixes #301
Outcome

Before
After

@gnprice could you review this PR and let me know if any changes are required.

When User.avatarUrl is null, we should handle that by computing a gravatar.

Ah, no, I've now gone and reread #255 and I see that's not the correct behavior here.
Currently there should not be any null values in User.avatarUrl. (There will be some after #254 and/or #255.) Therefore, if we see a null value there, we should just show SizedBox.shrink() as we do now.
If we were to turn on both of the features described in #254 and #255 (and not do anything else), then we'd have two different cases both show up as null there: the case where we should compute a gravatar, and the case where we should compute a fallback avatar URL. That's because one of those would be represented by "avatar_url": null in the JSON, and the other by avatar_url being absent in the JSON. Both of those conditions are, by default, rendered as avatarUrl: null by Dart's JSON deserializer. (Which is a good thing — it stems from the fact that Dart does not share JS's distinctive bad design choice of having two different values undefined and null.)
When we do go to take care of those two issues, then, we'll need to add a wrinkle in the User class to represent that distinction. That's all out of scope for this issue, so for this PR let's leave out both the gravatar case and the fallback case.

Sure, thanks for the clear questions.
When User.avatarUrl is null, we should handle that by computing a gravatar. That doesn't have to be part of this PR, though, as it's a separate issue we're tracking as #255.
There's another case where resolvedUrl is null, though: if User.avatarUrl is a string that doesn't parse as a URL. In that case we should continue to return SizedBox.shrink(), just like if the user wasn't found. I think the cleanest way to fix that will probably be to push the tryResolveUrl call inside AvatarURL.fromUserOrBotData (note there's a free-standing function tryResolveUrl, so you don't need to pass the store around in order to do that).
For the size, the AvatarImage widget hasn't previously called for a size because it hasn't previously had anything it would do with that information — it's been always fetchinig the same image regardless of what size it'll be shown at. Now that it does need to know the size, though, it should take the size as a parameter.
I don't think there's a good default that AvatarImage can use for the size — sometimes it'll be used in a place where it's going to be shown large, and sometimes where it's going to be shown small, and there's no reasonable way it can guess. So the parameter should just be required.

@gnprice You mentioned that size should be a required parameter for AvatarImage, but in the (recent_dm_conversations.dart)[https://github.com/zulip/zulip-flutter/blob/main/lib/widgets/recent_dm_conversations.dart] file AvatarImage is used without providing the size param.

This is in the previous version of the app. How should this be handled !

At those call sites, the caller should pass the actual size the AvatarImage will have.
If you look around at the surrounding code, the caller does know the size. It just hasn't been passing it to AvatarImage because there hasn't previously been anything that AvatarImage would do with that information.

At those call sites, the caller should pass the actual size the AvatarImage will have.
If you look around at the surrounding code, the caller does know the size. It just hasn't been passing it to AvatarImage because there hasn't previously been anything that AvatarImage would do with that information.

Refer commit 533c6ca

@gnprice could you review the changes! Thank you for your efforts 🎉

@gnprice I've made the suggested changes, do let me know if any revisions are required. I think the CI is currently stuck, is it possible for you to manually restart it!

@gnprice I've made the changes, if it's alright then I will start squashing the commits as requested

Sure — please go ahead.

Thanks for the revision! This is getting a lot closer.
There's a variety of comments below. One other important thing we'll need before we can merge this is that the commits in the branch need to be squashed and reorganized into clear and coherent commits, following the Zulip project's Git style: https://zulip.readthedocs.io/en/latest/contributing/commit-discipline.html
For this PR, I think that will mean two commits:

One commit adding size to AvatarImage, and making the necessary changes where that widget is used.
Then one commit with the rest of the changes: adding AvatarUrl and subclasses, using that from AvatarImage, adding the tests.

To get there from your current structure, my suggestion would be:

First, git rebase to get on top of current main and eliminate the merge commit. (If this hits any merge conflicts, the easiest solution might be to abort the rebase and use git merge origin/main instead. You'll still have conflicts but all at once, which in this case is probably easiest.)
Then git reset origin/main, so that your current commit is origin/main but you have all your changes to files.
Then git add -p followed by git commit, to make the first commit.
Then git add and git commit to make the second commit.

But if you have another procedure you're comfortable with that gets the right result, that's good too. If you have any questions with the Git manipulations or want any help, just ask in #git help on chat.zulip.org.

you mentioned there should be a separate NFC commit for the AvatarImage size property, but I would need the size property to compute the physicalSize in content.dart, also in the content_test.dart the size prop is required for the tests. Hence I'm currently adding all the changes in a single commit. Let me know if there is some other workaround!

@gnprice I've made the suggested changes!

Looks good — merging. Thanks @saisumith770 for all your work on this!


Showed warning for some of unmatched column in var types but not all

due to invalid ascii characters at bottom lines


Doc Feedback: Forget password screenshot uses old UIs

Location : https://apim.docs.wso2.com/en/latest/Administer/ProductSecurity/identity-management-for-the-api-store/
Now in APIM 3.0.0, the default login screen is following.

But the screenshots on the page show different(old UIs). Better to use the default UIs since the majority of the users are going to stick to that.

This has been fixed in [1]
[1] https://apim.docs.wso2.com/en/latest/Administer/ProductSecurity/identity-management-for-the-api-dev-portal/


Added derivative of leaky Relu

References to other Issues or PRs or Relevant literature

Fixes #60
Brief description of what is fixed or changed
Created LeakyRelu class with activation and derivative methods. The derivative method returns the array of derivatives of the applied Leaky Rectified Linear Units on X Vector.

@agrawalshubham01 I requested to do it separately


round-val =0

this unsets round-val ... need to have this as a real setting and use defined-ness to turn off

on reflection this is cool

can set to 1 for nothing right of the decimal point
can set to 0 to reset to apply no rounding


Different sequence length in one batch

I didn't see attention_mask in the forward, so if the batch length is inconsistent in the training process. How can I fix it.

I don't think you need to set an attention mask because for the causal attention, padding tokens at the right of the sequences will not affect the training result.

I don't think you need to set an attention mask because for the causal attention, padding tokens at the right of the sequences will not affect the training result.

The padding token is at the end of the sequence, but based on the provided code, the memory token is positioned to the right of the padding token. If the input sequence lengths are inconsistent, the padding token could affect the attention score of the memory token, right?


slow response times

Describe the bug
The cli is quite slow at responding to commands because of #29 and only on windows
To Reproduce
Run is --help, it's slow
Expected behavior
It should be running much faster.

Closing as it's only in the help command and it's just due to slow json parsing.


Sharing Bug Fix

Fixes #2057.
Our analysis assumed that state would not live beyond a single invocation (similar concept to the infer-share pass). But as #2057 shows, if a component's state lives beyond a single invocation, then our live range analysis should account for this.
The solution is to do exactly what we do for while/repeat loops.

Amazing!

Super, thanks a bunch! I've glanced over the changes, mostly to learn, and am happy for you to merge when ready.

Can we do specific analysis on components to determine if they expose any live-after-invoke state (similar to domination)?

I was thinking about this, and I think the same analysis as infer-share should work. Here's my thinking: we were running into problems because we were still observing the state of a cell that had lived on from a previous invocation. But if every read of a cell is dominated by a write to that same cell, then it is impossible for us to observe state from a previous invocation.
On a related note, I'm beginning to realize that "every read of a cell is dominated by a write to the same cell" seems like it could be accomplished using live range analysis (something like, if there are no cells live at the "entry" node of the control, then we know that every read of a cell is dominated by a write to the same cell-- if there were a read that wasn't dominated by a write, then that cell would be live at the entry node).


Cannot add custom repo

Describe the bug
When I'm adding the custom repo according to the docs I get in Installation Step 4:
No RepositoryFile.MAINIFEST_JSON file found 'custom_components/hoymiles_wifi/RepositoryFile.MAINIFEST_JSON'
popup at the top of the custom repo list
To Reproduce
Steps to reproduce the behavior:
As described in the readme of the repo, happens in step 4
Expected behavior
Custom repo to be added from github
Screenshots
Desktop (please complete the following information):

OS: OSX
Browser Safari
HA: 2024.1.3
HA Host: ODROID

I tried with latest and home assistant core and could not reproduce the issue. Can you maybe provide some more log information from HACS install process?

ok, looks like HACS needed an update. Might be worth mentioning, as HACS is never updated unless you manually run the installer from the terminal again. It seems versions from 2022 are broken with today's HA.
Also some instructions on your doc are outdated, eg. step 5, it never asks to 'save' or 'install', but 'download', etc.
cheers

solved

Thanks for letting me know. Will update the Readme.


Using Merchant Almanac intermittently causes game to crash

Minecraft version & Mod Version: 1.12,; 1.7.10
Single player or Server: Single Player
Describe problem (what you were doing / what happened): Normal use of the Merchant Almanac will randomly crash the game. Seemingly when using it rapidly.
Crash log (if any):
(If this is a suggestion and not a bug report, delete all this and just write your suggestion!)

I will look into fixing this.  Do you have a crash log?

will be in the /crash-reports/ folder

I think i fixed it. not sure exactly without crash log but version 1.8.0 will be out soon

Closing due to no response and no crash log.  As far as I know the fix i put in 1.8.0 worked https://minecraft.curseforge.com/projects/cyclic/files/2476006

I had the Merchant Almanac crash my server a few times. https://pastebin.com/UPcZmgMY It looked like if I was standing too close to solid blocks when doing the trades it would crash.

I just got a crash that looks an awful lot like the one mrdeadlocked got.
crash-2017-10-20_14.44.12-server.txt


ci(*): add eslint-plugin-react-compiler to prepare react 19

related #929
PR Checklist

[x] I did below actions if need

I read the Contributing Guide
I added documents and tests.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 83.44%. Comparing base (2594ab3) to head (c6c1137).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #934   +/-   ##
=======================================
  Coverage   83.44%   83.44%
=======================================
  Files          33       33
  Lines         435      435
  Branches       99      100    +1
=======================================
  Hits          363      363
  Misses         62       62
  Partials       10       10

Components
Coverage Δ

@suspensive/react
96.98% <100.00%> (ø)

@suspensive/react-query
13.79% <ø> (ø)

@suspensive/react-await
100.00% <ø> (ø)

@suspensive/react-image
23.52% <ø> (ø)


fix address form country state label for

1. Why is this change necessary?
In the file "address-form.html.twig" is a wrong input id defined in the country state label.
2. What does this change do, exactly?
Fix the country state label "for" by set it to the country state id.
3. Describe each step to reproduce the issue or behaviour.
Just take a look in the code.
4. Please link to the relevant issues (if any).
5. Checklist

[ ] I have written tests and verified that they fail without my change
[ ] I have created a changelog file with all necessary information about my changes
[ ] I have written or adjusted the documentation according to my changes
[ ] This change has comments for package types, values, functions, and non-obvious lines of code
[ x] I have read the contribution requirements and fulfil them.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Hello,
thank you for creating this pull request.
I have opened an issue on our Issue Tracker for you. See the issue link: https://issues.shopware.com/issues/NEXT-22122
Please use this issue to track the state of your pull request.


added minio example for 1.11

update guide for minio with dcos 1.11
issue #303

Good clarification with regard to editing the service, thanks for the fix-ups!


parser: support IEEE Inf and NaN

Closes #3702

Looks good to me.  I'll let @nwt approve in case he thinks of a corner case.


Need flask if you want to use the dectris sim

Contributor Checklist:

[ ] I have added or updated my entry in the creators.json file
[ ] I have added a changelog entry for my contribution
[ ] I have added/updated documentation for all user-facing changes
[ ] I have added/updated test cases

Reviewer Checklist:

[ ] /azp run libertem.libertem-live-data passed

/azp run libertem.libertem-live-data


[Bug]: KafkaIO considers readCommitted() as it would commit back the offsets, which it doesn't

What happened?
I have been reading from Kafka and trying to figure out which offset management would be the best for my use-case. During that I noticed something odd.
https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2359-L2362
private boolean configuredKafkaCommit() {
  return getConsumerConfig().get("isolation.level") == "read_committed"
      || Boolean.TRUE.equals(getConsumerConfig().get(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG));
}

https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2292-L2298
https://github.com/apache/beam/blob/c9c161d018d116504fd5f83b48853a7071ec9ce4/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L2321-L2334
The name of the method, and how it's being used in the code certainly suggest that using read_committed isolation level handles and commits kafka offsets.Seemed strange, but I'm not a Kafka pro, so let's test it. Well it does not.

using ONLY ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG does commit it

using ONLY commitOffsetsInFinalize() does commit it

using ONLY withReadCommitted() does NOT commit it

Dataflow, 2.40.0 Java SDK, without explicitly enabling SDF-read
So is it a bug, or what am I missing here?
If it is indeed a bug, then is it with the read_committed (so it should commit it although found no explicit documentation about that anywhere), or having that isolation level shouldn't prefer the commit in the finalize and that method is wrong?

@johnjcasey:
withReadCommitted() doesn't commit messages when read, it instead specifies that the kafka consumer should only read messages that have themselves been committed to kafka.
Its use is for exactly once applications.
@johnjcasey
Which looking at your message again, would imply that the configuredKafkaCommit() method shouldn't inspect isolation.level
Issue Priority
Priority: 2
Issue Component
Component: io-java-kafka

.take-issue


Add audio engine v2

This change adds the initial audio engine v2 implementation and introduces the API for sound and streaming sound creation and playback.

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

Snapshot stored with reference name:
refs/pull/15839/merge
Test environment:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/index.html
To test a playground add it to the URL, for example:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/index.html#WGZLGJ#4600
Links to test babylon tools with this snapshot:
https://playground.babylonjs.com/?snapshot=refs/pull/15839/merge
https://sandbox.babylonjs.com/?snapshot=refs/pull/15839/merge
https://gui.babylonjs.com/?snapshot=refs/pull/15839/merge
https://nme.babylonjs.com/?snapshot=refs/pull/15839/merge
To test the snapshot in the playground with a playground ID add it after the snapshot query string:
https://playground.babylonjs.com/?snapshot=refs/pull/15839/merge#BCU1XR#0

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgpuplaywright/index.html

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgl2playwright/index.html

Visualization tests for WebGL 1 have failed. If some tests failed because the snapshots do not match, the report can be found at
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgl1/index.html
If tests were successful afterwards, this report might not be available anymore.

WebGL2 visualization test reporter:
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgl2playwright/index.html

Visualization tests for WebGPU (Experimental)
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgpuplaywright/index.html

Visualization tests for WebGL 1 have failed. If some tests failed because the snapshots do not match, the report can be found at
https://snapshots-cvgtc2eugrd3cgfd.z01.azurefd.net/refs/pull/15839/merge/testResults/webgl1/index.html
If tests were successful afterwards, this report might not be available anymore.


Setting EmptyTemplate on CompletionList does not have any effect

The issue is the same as described in http://www.codeproject.com/Messages/4980791/Usage-of-CompletionList-EmptyTemplate.aspx:
Setting the EmptyTemplate on the CompletionList of a CompletionWindow does not have any effect. I intend to use code like the following to define the template:
<Style TargetType="CodeCompletion:CompletionList"
       BasedOn="{StaticResource {x:Type CodeCompletion:CompletionList}}">
    <Setter Property="EmptyTemplate">
        <Setter.Value>
            <ControlTemplate>
                <Label Content="No suggestions"/>
            </ControlTemplate>
        </Setter.Value>
    </Setter>
</Style>

As "Member 10819996" wrote:

The template does not show up, neither when the CompletionData list is empty from start, nor when it initially contains items, but all items get filtered out.

may be you can try changing to   to

May be you can try changing <ControlTemplate> to <ControlTemplate TargetType="avalonEdit:CompletionList">


Error Message: Uncaught (in promise) invalid stored block lengths

I've written some code for processing uploaded files using NextJS and Node14.   When running locally all works well however when I deploy to Vercel I receive the error message:
Uncaught (in promise) invalid stored block lengths

Its still Node14 when deployed and happens in Chrome.
Any ideas?

That error suggests some sort of corruption in the bytes.  Azure and AWS automatically do some UTF8 conversion which causes issues with XLSX files.  https://github.com/SheetJS/sheetjs/tree/master/demos/function#aws-lambda discusses the workaround of adding "Binary Media Types".  Vercel may require a similar setting.

Please verify the actual file data is correct with a checksum.  If the data is what you expect and the original file opens properly in Excel, please share the file and the issue will be reopened.

having same issue when deploy using using pm2. processing xlsx no error on localhost but got same error when in server

@h4ckm03d probably some network issue. AWS needs some binary setting

@reviewher the problem is not network issue, it caused by swcMinify is true on next.config.js. After I disabled swcMinify then rebuild. everything run normally.

That sounds like a potential bug with swc? https://github.com/swc-project/swc/issues


Linting sweep with gersemi

Running gersemi over entire repo as it hasn't been done yet. No functional changes, just linting

Thanks for doing this, and appologies for the hassle!

Related to #47


Research PHP

Want to use PHP as the language for the webpage. For example, will have an Index.php
ToDo:

[ ] Learn some basic syntax of PHP and try to make a hello world page that can be hosted on the PiServer.

PHP Works well with MySQL or MariaDB. Use them in conjunction.

Good resource to reference:
https://pimylifeup.com/raspberry-pi-apache/

hello world successfully hosted


Hunterize vectorial

@ruslo

Released: https://github.com/hunter-packages/vectorial/releases/tag/v0.0.0-ae7dc88-p0


readme: adjust timeline [skip ci]

Hi @web2033 and thank you for the PR!
A similar fix has been merged recently, so I'm going to close this one.
See de7f9d1efd7fa19a908357d3f3a706c52694d8bd


Unable to reproduce LLaMA-7B results when training from scratch

Hi,
I was trying to reproduce the LLaMA-7B with 1 gist token results from scratch following the training instruction in the README. I ran the script below on 4 A100-80GB GPUs:
TAG="train80g"

port=$(shuf -i25000-30000 -n1)

deepspeed --master_port $port --num_gpus=4 --no_local_rank \
    --module src.train \
    +model=llama-7b wandb.tag=$TAG \
    training.deepspeed=ds_configs/stage3.json \
    training.gist.condition=gist \
    training.gist.num_gist_tokens=1

However, the final results after 3 epochs are much lower than the reported ones in the paper. I got seen 51.24, unseen 42.01, human 19.00 for ROUGE-L. I tried training for longer epochs but it didn't help with unseen and human ROUGE-L results. I did not change anything in the training config other than the wandb account.
I also evaluated the 3 provided checkpoints (gist, pos_control, neg_control) and the results are consistent with the paper (< 0.1 difference in terms of ROUGE-L) for all of them, so the evaluation code should function normally. Could you help double check if the above training setup is correct, and do you have any suggestions on how to reproduce LLaMA results in the paper?

Hello, this is unrelated to your issue but I was hoping you could point me in the right direction since I'm an ML beginner trying to leverage this repo for another organization.
Did you try running compress.py on your GPU set up? Did you have to make any changes to the file's code apart from the device_map="auto"/ device_map="balanced" parameter passed into the model initializer?
Thanks.

Hello, this is unrelated to your issue but I was hoping you could point me in the right direction since I'm an ML beginner trying to leverage this repo for another organization.
Did you try running compress.py on your GPU set up? Did you have to make any changes to the file's code apart from the device_map="auto"/ device_map="balanced" parameter passed into the model initializer?
Thanks.

Hi, I tried the example command in the README and it works without an issue:
python -m src.compress --model_name_or_path llama-7b-gist-1-recovered \
    --instruction "Name the top cities in France that should not be missed. Include the best aspects of each place as well."

where llama-7b-gist-1-recovered is the path of the llama checkpoint after applying the llama-7b-gist-1 weight_diff. I did not modify the code at all.
Also, I would recommend you open a separate issue if there's any follow-up / new question.

Hi, give me a few days to look into this. Have you tried training the positive control model? Curious if the issue is just the gist model or all of the models.

Also, are you using the decapoda-research llama checkpoint? Can you check whether it might be due to an incorrect tokenization config with the decapoda models? See

https://huggingface.co/decapoda-research/llama-7b-hf/discussions
https://github.com/tloen/alpaca-lora/issues/279

Hello, this is unrelated to your issue but I was hoping you could point me in the right direction since I'm an ML beginner trying to leverage this repo for another organization.
Did you try running compress.py on your GPU set up? Did you have to make any changes to the file's code apart from the device_map="auto"/ device_map="balanced" parameter passed into the model initializer?
Thanks.

Hi, I tried the example command in the README and it works without an issue:
python -m src.compress --model_name_or_path llama-7b-gist-1-recovered \
    --instruction "Name the top cities in France that should not be missed. Include the best aspects of each place as well."

where llama-7b-gist-1-recovered is the path of the llama checkpoint after applying the llama-7b-gist-1 weight_diff. I did not modify the code at all.
Also, I would recommend you open a separate issue if there's any follow-up / new question.

I'm guessing why it worked with no modification was because your GPU specs fit the required ones perfectly haha. Thanks

Hi, give me a few days to look into this. Have you tried training the positive control model? Curious if the issue is just the gist model or all of the models.

Thank you for the reply. I haven't tried training the positive control model but let me do it now. Just another two quick follow-ups:

Is there anything else that needs to be changed than setting training.gist.condition=pos_control to train the positive control model using the script above?
I noticed that although the config specifies a cosine lr scheduler, the logged wandb lr was always unchanged. I am not sure if this is a wandb issue (as I have encountered similar logging issues in the past) or the lr scheduler was truly not working properly.

Also, are you using the decapoda-research llama checkpoint? Can you check whether it might be due to an incorrect tokenization config with the decapoda models? See

https://huggingface.co/decapoda-research/llama-7b-hf/discussions
What are the eos_token_id and bos_token_id tloen/alpaca-lora#279

This is a good point. All my previous experiments were done using the decapoda-research checkpoint, and I was actually redoing experiments with the official checkpoint today. I'll let you know how it goes.
I also have a question w.r.t. the provided gist checkpoint: it seems like in the generation_config.json, the tokenization looks like this:
{
  "_from_model_config": true,
  "bos_token_id": 0,
  "eos_token_id": 1,
  "pad_token_id": -1,
  "transformers_version": "4.28.0.dev0"
}

While in the decapoda-research checkpoint it is:
{
  "_from_model_config": true,
  "bos_token_id": 0,
  "eos_token_id": 1,
  "pad_token_id": 0,
  "transformers_version": "4.27.0.dev0"
}

and in the official checkpoint:
{
  "_from_model_config": true,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "pad_token_id": 0,
  "transformers_version": "4.28.0.dev0"
}

Could you explain the differences in your gist checkpoing tokenization config? Thank you.

It seems like there may be some versioning issues here...I think I wrote this code and converted the official Llama checkpoint back when there were still some outstanding tokenization issues in the huggingface repo (e.g. prior to https://github.com/huggingface/transformers/pull/22402); note the pinned transformers version in this codebase's requirements.txt is fb366b9a https://github.com/huggingface/transformers/blob/fb366b9a2a94b38171896f6ba9fb9ae8bffd77af/src/transformers/models/llama/tokenization_llama.py
The Llama checkpoint I used has the following configs:
generation_config.json
{"_from_model_config": true, "bos_token_id": 0, "eos_token_id": 1, "pad_token_id": -1, "transformers_version": "4.27.0.dev0"}

config.json
{"architectures": ["LLaMAForCausalLM"], "bos_token_id": 0, "eos_token_id": 1, "hidden_act": "silu", "hidden_size": 4096, "intermediate_size": 11008, "initializer_range": 0.02, "max_sequence_length": 2048, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 32, "pad_token_id": -1, "rms_norm_eps": 1e-06, "torch_dtype": "float16", "transformers_version": "4.27.0.dev0", "use_cache": true, "vocab_size": 32000, "_name_or_path": "llama-7b"}

which seems to be incorrect, however, in a debug session after loading the LlamaTokenizer from this checkpoint, I get
ipdb> p tokenizer
LlamaTokenizer(name_or_path='llama-7b', vocab_size=32000, model_max_length=1000000000000000019884624838656, is_fast=False, padding_side='left', truncation_side='right', special_tokens={})
ipdb> tokenizer.bos_token_id
1
ipdb> tokenizer.eos_token_id
2

so I'm definitely a bit confused here, the pinned version of transformers I used might be overwriting the checkpoint values. In fact I recall having to manually overwrite the generation config here: https://github.com/jayelm/gisting/blob/297907c0414538c02c5ccb823f5dfa4de23f0efd/src/trainer_seq2seq.py#L303-L313
I previously attributed this to a DeepSpeed bug (for some reason I remember not seeing this issue without deepspeed) but maybe it's related to tokenization.
Some answers that would help clarify things:

Can you confirm you are using the same version of Transformers I am using?
Can you check what your bos and eos token ID are set to after loading the tokenizer in the training script when you train the model that observed degraded performance? Is it 1 and 2 (as it should be), or 0 and 1 (as in the decapoda-research checkpoint)?
I suspect this is a tokenization issue, but it might not be. If your pos control training run also exhibits degraded performance, then that would lean in favor of tokenization being the issue. But if it doesn't, then the issue is probably somewhere else, so I'm interested in what you find.

Appreciate your help debugging this!

Is there anything else that needs to be changed than setting training.gist.condition=pos_control to train the positive control model using the script above?

No, that is the only change required!

I noticed that although the config specifies a cosine lr scheduler, the logged wandb lr was always unchanged. I am not sure if this is a wandb issue (as I have encountered similar logging issues in the past) or the lr scheduler was truly not working properly.

Aha, this was also the case in the original Alpaca codebase—it specified a cosine LR scheduler but the LR wasn't actually changed at least according to wandb. I didn't look too closely into this. So even if not entirely correct, this is expected, and I observed this in my experiments.

Thank you for putting the efforts on the detailed response! With regard to your questions:

I was using 4.28.0.dev0, which was installed using requirements.txt. There is also a check_min_version("4.28.0.dev0") in your src/train.py. It should be the same as yours based on the commit number in requirements.txt.
My bos and eos token ID are correct -- they are 1 and 2, albeit the generation_config.json and config.json are wrong. I am not entirely sure, but I think this is because when loading the tokenizer by https://github.com/jayelm/gisting/blob/acd78b49111db30c3a24c32f625b85ae59934585/src/train.py#L140-L142
it will not look into files containing wrong IDs (generation_config.json and config.json), and will initialize bos_token="<s>", eos_token="</s>". These essentially correspond to 1 and 2 in the vocab (see https://github.com/huggingface/transformers/issues/22312). Thus the tokenizer seems to be loaded properly, and your manual overwrite fixes the generation & prediction. This makes the situation quite frustrating as I am not sure if the issue is with the tokenizer.
I am running pos control training with the hf checkpoint that I previously used now. I am also running gist and pos control training using the converted official checkpoint with the correct tokenizer config. I'll let you know how it goes!

I got no luck in running those new experiments:

pos control training with the decapoda-research checkpoint got seen 54.04, unseen 43.58, human 20.42.
pos control training with the converted official checkpoint got seen 53.77, unseen 43.75, human 18.80.
gist training with the converted official checkpoint got seen 51.26, unseen 42.32, human 18.89.

Basically there is no significant difference between the decapoda-research checkpoint and the converted checkpoint, and both gist and pos control experiments have worse results than those reported in the paper. Could you kindly consider rerunning the training on your side to verify if you're able to reproduce the results presented in the paper?

Ah, this could be the reason: I noticed that the python env I used for training is actually different than the python env described in requirements.txt. This is because I benchmarked the trained models using a different machine with a different python installation, and created requirements.txt from the python env I used for benchmarking, rather than the one I used for training.
I downloaded a fresh copy of the repo and reran the gist model training command with the python env below and reproduced the paper results exactly (see end of this post). So it's possible this might be the reason for the discrepancy (perhaps torch 2.0 -> torch 1.13)? If it's not too much trouble, could you try retraining with the following updated env:
Python version 3.10.8
requirements.txt generated from pipreqs: (you may have to install torch 1.13.1 from pytorch.org, and lmk if you have other trouble reproducing this env)
datasets==2.8.0
deepspeed==0.8.3
evaluate==0.4.0
fire==0.5.0
hydra-core==1.3.2
nltk==3.8
numpy==1.23.5
omegaconf==2.3.0
pandas==1.5.3
torch==1.13.1+cu116
torch_xla==1.0
tqdm==4.64.1
git+https://github.com/huggingface/transformers@fb366b9a
wandb==0.13.11

Somewhat confused that versioning could lead to such discrepancies, but if this is indeed the cause, sorry for the confusion! Even if this is not the issue, the fact that the positive control performance is also degraded on your end (and gist model still does well as pos control) indicates that the issue is not with the gist models specifically, but some mismatch with the general llama finetuning setup that we'd stil need to nail down.
Thanks again for your help!

Reproduction:

my job timed out before it finished eval on the unseen split at 3k steps, but I observed

seen 57.8,
unseen 46.2 (at 1.5k steps)
human 23.9

which is in line with the paper.

Also, on the off chance that we are still using different llama-7b checkpoints somehow, here are the shasums of the checkpoint I am using
da1ef16b048f5d759149c7472b6e4858f19bc287  config.json
e37700792e352efa5dba3c83478ff31d60b50e02  config_original.json
a2badb096591d9e0ca3f1c1151a4ac85063a6325  generation_config.json
0096c5b09f6403a1a1a211a50090d37d24acbd12  pytorch_model-00001-of-00033.bin
83af20eb30fb7b52e62e80a07cb8a7de2c35612e  pytorch_model-00002-of-00033.bin
1b73432827a81a77a5548652ee6fe1f127d53aeb  pytorch_model-00003-of-00033.bin
98cde125792b2cf172581f26db3e3583765bd07f  pytorch_model-00004-of-00033.bin
3eecc843ced51c6b8f24ba5e81a33c28c0028abb  pytorch_model-00005-of-00033.bin
1cbd688ddd5411443fad88a6ac32656f74ba66d8  pytorch_model-00006-of-00033.bin
021af212e30f8294e24996eb9c3710ab5963d93f  pytorch_model-00007-of-00033.bin
099abf28dc554ac030db312e518335f47dbc630d  pytorch_model-00008-of-00033.bin
4d9af68d697c3a9a6624d69f32dfc5eaa4f02b89  pytorch_model-00009-of-00033.bin
7358c017a646ac18331a1b318a9bea6f87c505e3  pytorch_model-00010-of-00033.bin
eae98416c3f8ff31a532ac1b017183c610c07916  pytorch_model-00011-of-00033.bin
f4fd5e8087ddb9b46ecd729a7e3970711233b2d3  pytorch_model-00012-of-00033.bin
f3a296be997c73e5f39c7b6bc5b534840a8ed8f0  pytorch_model-00013-of-00033.bin
e98055c932e250352b0660b9c47889af2829a645  pytorch_model-00014-of-00033.bin
072af56043c00bf9c1d30b84d79612feaf4660b4  pytorch_model-00015-of-00033.bin
8ac62fb6884080851c3cfffc151b8aa13b6cb919  pytorch_model-00016-of-00033.bin
e3679a8b6e64dff5b8163e1428658a4ed1428e67  pytorch_model-00017-of-00033.bin
2fb9c4413cbcfc6be5b94229b38c40781f43a41b  pytorch_model-00018-of-00033.bin
084b9637913276c7ce9f8d92f1a80a5b89eb9146  pytorch_model-00019-of-00033.bin
7158b438698dc10235212bf609fa5ccb74145109  pytorch_model-00020-of-00033.bin
bfbc357f4c5db9e07ef0baa29563c648a17d463d  pytorch_model-00021-of-00033.bin
cdcf5c860eeab5632113dd39260e06ae5aa8e301  pytorch_model-00022-of-00033.bin
39a24a8576ea63fc238f26dd6dc599a3e9efcd11  pytorch_model-00023-of-00033.bin
bd8222ced6951461781c5fb45c7c7c63ae0a6e6b  pytorch_model-00024-of-00033.bin
2168e992c91dcadbc5ba8c9de9f1f5c302971ae4  pytorch_model-00025-of-00033.bin
7e657df1760b10db893f76835ec524c633f1da53  pytorch_model-00026-of-00033.bin
aed1f083e57dc8e711ee3f50b06c2d36a466bc18  pytorch_model-00027-of-00033.bin
8e0bcdb0d1b974fe9967c638e4c24d6172a09402  pytorch_model-00028-of-00033.bin
c77a739a98c877616d7d5e65991cc893d5988bc2  pytorch_model-00029-of-00033.bin
c4b9df5137840cafafd26de980f55bf0f907c281  pytorch_model-00030-of-00033.bin
37022a5d1d8e71a0b5a84f6e51d3f1454ec48beb  pytorch_model-00031-of-00033.bin
9e789f6bbe3fb919c9eee6f6ad47818f1747ea74  pytorch_model-00032-of-00033.bin
1444445a36623906176a8e636dc1c51763f3c05d  pytorch_model-00033-of-00033.bin
16b8de6e0ac1e2b75d1032c8315c3df2a115f0d0  pytorch_model.bin.index.json
bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f  special_tokens_map.json
39d4ad6eced3ea31d27f762010a5ba67deb8b3ff  tokenizer_config.json
7a4e789beca293352e60b6fad5eef1908070cee0  tokenizer.model

Since you've tried a few llama models, it seems more likely that the discrepancy is due to the codebase, but just in case, you can check these hashes. I can also email you the llama checkpoint directly if you'd like—you can reach out to my academic email (available on my website) and we can sort something out. But it's probably worth synchronizing python envs first.

Thank you for the thorough investigation. Let me run experiments with your newly provided environment first. If that doesn't work I'll dig into the checkpoint.
Just a side note, I was actually running most of my experiments using 8 A100-40GB gpus since I have relatively limited A100-80GB resources. I changed per_device_train_batch_size and per_device_eval_batch_size from 4 to 2, so the batch size would still be 128 and theoretically there shouldn't be any difference between this setup and the suggested setup. Nonetheless, I will verify if I can reproduce the results with any of the setups.

Switching to the provided python envs indeed fixed the training! I got seen 57.39, unseen 46.53, human 25.83 for the 1 gist token setting trained using the converted official checkpoint. These are pretty much consistent with the paper results.
It requires a bit more changes to the requirements.txt to set up the environment. torch==1.13.1+cu116 could not be found by pip, and I first installed pytorch via
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia

Then slightly updated the requirements.txt to
accelerate==0.18.0
datasets==2.8.0
deepspeed==0.8.3
evaluate==0.4.0
fire==0.5.0
hydra-core==1.3.2
nltk==3.8
numpy==1.23.5
omegaconf==2.3.0
openai==0.27.2
rouge_score==0.1.2
sentencepiece==0.1.98
pandas==1.5.3
tqdm==4.64.1
git+https://github.com/huggingface/transformers@fb366b9a
wandb==0.13.11

Using this environment could successfully reproduce the gist results when training from scratch on my end. I think it's worth further investigating the version mismatch of which packages caused the performance degradation in my previous experiments, but this can serve as a temporary solution for now. Thanks again for your time in looking into this!

I ran a few more experiments and located the issue -- the performance discrepancy is not caused by the torch version, but the deepspeed version. I was still able to reproduce the paper results after changing the torch version back to 2.0.0, so I did some further debugging and realized that since I did not create a new env from scratch previously, my older experiments were run with deepspeed 0.9.4 instead of deepspeed 0.8.3. Somehow pip install -r requirements.txt did not overwrite the existing deepspeed installation on both of my A100-40G servers and A100-80G servers...
I then conducted experiments using the same env with only the deepspeed versions being different, and verified that only the 0.9.4 one resulted in the worse performance that I encountered earlier in this issue.
I haven't figured out why the up-to-date deepspeed could cause such performance degradation. And I apologize for my oversight (again!) -- your current listed dependencies should be all good.

Apologies for the late response, I've been on vacation 😄
Glad to hear the results replicate, and thanks so much for looking so closely into this! This will help a lot with reproducibility.
Super weird that the DeepSpeed version leads to such drastic performance differences. I'll make a note of this in the repo when I get back from vacation.

Hi @Xiuyu-Li, could you please share your reproduced results? I also tried to replicate Llama-7b from scratch, and this is my results:
wandb: Run summary:
wandb:                   eval/gen_len 40.231
wandb:             eval/human_gen_len 88.93254
wandb:                eval/human_loss 2.18223
wandb:              eval/human_rouge1 28.2652
wandb:              eval/human_rouge2 10.6288
wandb:              eval/human_rougeL 23.9176
wandb:           eval/human_rougeLsum 26.8962
wandb:             eval/human_runtime 10497.9282
wandb:  eval/human_samples_per_second 0.024
wandb:    eval/human_steps_per_second 0.002
wandb:                      eval/loss 1.25835
wandb:                    eval/rouge1 52.6178
wandb:                    eval/rouge2 29.9502
wandb:                    eval/rougeL 46.6212
wandb:                 eval/rougeLsum 49.1622
wandb:                   eval/runtime 5599.2181
wandb:        eval/samples_per_second 0.179
wandb:              eval/seen_gen_len 22.35
wandb:                 eval/seen_loss 0.58948
wandb:               eval/seen_rouge1 59.6561
wandb:               eval/seen_rouge2 34.5262
wandb:               eval/seen_rougeL 57.779
wandb:            eval/seen_rougeLsum 58.4709
wandb:              eval/seen_runtime 2058.5075
wandb:   eval/seen_samples_per_second 0.486
wandb:     eval/seen_steps_per_second 0.031
wandb:          eval/steps_per_second 0.011
wandb:            eval/unseen_gen_len 40.231
wandb:               eval/unseen_loss 1.25835
wandb:             eval/unseen_rouge1 52.6178
wandb:             eval/unseen_rouge2 29.9502
wandb:             eval/unseen_rougeL 46.6212
wandb:          eval/unseen_rougeLsum 49.1622
wandb:            eval/unseen_runtime 5602.1366
wandb: eval/unseen_samples_per_second 0.179
wandb:   eval/unseen_steps_per_second 0.011
wandb:                    train/epoch 3.0
wandb:              train/global_step 3000
wandb:            train/learning_rate 2e-05
wandb:                     train/loss 0.3331
wandb:               train/total_flos<PHONE_NUMBER>1568.0
wandb:               train/train_loss 0.6486
wandb:            train/train_runtime 69016.7382
wandb: train/train_samples_per_second 5.567
wandb:   train/train_steps_per_second 0.043

I train the model on 4*A100(80G) which takes about 24 hours to finish training and I use deepspeed==0.8.3 and torch==1.13.1+cu116.

Hi @Hannibal046 , the results you just reported align with the paper's, namely these lines:
wandb:              eval/human_rougeL 23.9176
wandb:               eval/seen_rougeL 57.779
wandb:             eval/unseen_rougeL 46.6212

and the table:

so it seems like you've successfully reproduced the results. Did you have other questions?

Hi, @jayelm
I don't have further questions and I appreciate the great idea the great code, from which I learned a lot!

Hi @jayelm, sorry to bother.
I found a weird chart in the wandb panel with respect to train/learning_rate.

I expected it to be a cosine shape with warmup considering that config:

Hi @Hannibal046, see this comment earlier in the thread:

3. I noticed that although the config specifies a cosine lr scheduler, the logged wandb lr was always unchanged. I am not sure if this is a wandb issue (as I have encountered similar logging issues in the past) or the lr scheduler was truly not working properly.

Aha, this was also the case in the original Alpaca codebase—it specified a cosine LR scheduler but the LR wasn't actually changed at least according to wandb. I didn't look too closely into this. So even if not entirely correct, this is expected, and I observed this in my experiments.

aha, got it! my mistake and appreciate your help!

Hi @Hannibal046 , those are the things I'm aware of! First step would be migrating to a newer huggingface version while keeping the DeepSpeed version the same—you should be able to exactly reproduce the results. Second you can try debugging what's going on with DeepSpeed, I really have no clue and haven't had time to look myself.
If you do figure out either of these, please let me know and/or submit a PR, would be really helpful!


add an about/version info dialog

user asked if there is a way to tell what version he's running - there isn't really...

Added in https://github.com/fijam/gmdrec/commit/eec3af771b138473c6772c674094af2c7a2528aa


How to set credentials on windows

Not exactly sure how to set the credentials on windows
I tried using the credential manager but it requires a username does the websocket connection require one?

Hey,
if you enable authentication and set a password inside of the WebSocket server settings extension in OBS, you'll have to pass the password along for this extension to use.

The password can be set using the command "OBS Switcher: Set OBS password" (found in the Command Palette)

Which then will present a prompt for the password

Oh awesome thank you :)


[Question] How to address "/usr/bin/ld: cannot find -lodbc"?

I have the Db2 driver installed, cloned this git repo and ran the commands to build the example connect or the package. Both return the error message

= note: /usr/bin/ld: cannot find -lodbc
collect2: error: ld returned 1 exit status
error: aborting due to previous error
error: could not compile ibm_db

How can I address that error? I searched for libodbc.so / libodbc.a but could not find it. This is on RHEL 8.3

Hi, @data-henrik not sure why the package is looking for odbc.
Can you try installing using crates.io.
Also make sure that LD_LIBRARY_PATH is set i.e.
export LD_LIBRARY_PATH="/foo/bar/clidriver:/foo/bar/clidriver/include:/foo/bar/clidriver/bin:/foo/bar/clidriver/lib"
Let me know if above works.

cargo run --package ibm_db --example setup
That's the failing command.

Hi @data-henrik I reinstalled the package on RHEL 8 and it installed successfully.
I copied the src, example, cargo.toml and build.rs files on the machine and made sure that LD_LIBRARY_PATH is set to point to the correct directory of CLIDriver i.e. db2 driver in the format mentioned in prev message.
Then I ran the command "cargo run --package ibm_db --example setup" and it ran seamlessly without error.
Can you please troubleshoot a bit more and check if some other settings are not missing like gcc etc.
Some other dependent package might have some dependency that is not getting satisfied possibly.
ibm-db package has dependency only on db2 library.
Make sure you have the CLI driver installed. If not simply run the setup.rs file to download the same for your environment.

@data-henrik ,
I wanted to let you know that I also have issues with the environment variables not being used as lookup directories for the linker. I'm not sure why.
I resolved my issue by using this command:
RUSTFLAGS='-L /db2-query/clidriver/lib' cargo build --release
replace with your correct driver's lib path.
Hope it helps! 👍


GSNGM-644: Geometry filters (+ GSNGM-661)

great! Can we make the background color of this button group normal-panel-grey (#f1f3f5) and not the very-light grey? (while keeping the very light grey for the geometries)

great! Can we make the background color of this button group normal-panel-grey (#f1f3f5) and not the very-light grey? (while keeping the very light grey for the geometries)

fixed


Feature request: Support merging in-memory tables

Having the consuming code provide a utf-8 stream of some sort would allow for on-disk, network, in-memory etc instead of forcing files to be read from disk.

Ehm what?

Merging only supports reading from disk as I understand it but allowing a consuming library to merge from other sources (MemoryStream, etc) would be useful.

v0.8.0 of Nett.Coma allows to use custom store implementations. By implementing a small interface it should be quite easy to use s stream based TOML stores.


feat: add myself to robozome and reorder list to alphabetical order

[x] Reordered list of owners to alphabetical order for better readability
[x] Added myself as both approver and reviewer
[x] removed @ tssala23 since he is not going to work on this project anymore

Part of #17

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign larsks after the PR has been reviewed.
You can assign the PR to them by writing /assign @larsks in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @tumido

@mimotej Hello, I will still be working in Thai despite my internship finishing. Just had to get through the first week of classes and sort out my schedule.

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: tumido
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [tumido]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/hold

lol

@mimotej can you please reintroduce @tssala23 back again? Prow doesn't respect its own labels.. :smile:


Tournament tscheck

passes tscheck, and is tested.

Made a few changes to simplify stuff. I've tested it but I want you to test it as well and give me your thoughts

Tested, works great just had to fix the replies for the participant command to be edits of a deferred reply.


Toolbar subtitle isn't inflated right away

I ran into an issue where rarely the subtitle of my fragments wouldn't be using the correct font.
The issue seems to be that since I was using fragment transactions and since the first fragment in the activity didn't have a subtitle, the subtitle view was never added to the toolbar. (At least I think this is why since the Toolbar will only have one child in the callback from onGlobalLayout of CalligraphyFactory line 162) Then when I later switch in a fragment that has a subtitle the textview for it is created and Calligraphy doesn't catch it. A workaround I found for it is commenting out the following from CalligraphyFactory
// Our dark deed is done
if (Build.VERSION.SDK_INT < Build.VERSION_CODES.JELLY_BEAN) {
//noinspection deprecation
parent.getViewTreeObserver().removeGlobalOnLayoutListener(this);
} else {
parent.getViewTreeObserver().removeOnGlobalLayoutListener(this);
}

But that goes back to #121 and #102 which isn't ideal. Is there a better way to accomplish this? I could potentially get the toolbar view, find the textview and set the font in these rare cases but I was hoping there would be an easy fix I just hadn't found yet that will allow Calligraphy to pick it up automatically
Other info:
This only started happening after switching from appCompat 19 to 22, I don't know if it happens on 21 but I can confirm if you think that'll help.
I'm using AppCompatActivity and I'm NOT setting the Toolbar and just letting the default one happen. I can try it the other way as well if you think that'll help

You could probably work around this by setting the Subtitle to an empty string (" "), I need to check the impli but that should instantiate the subtitle and let the listener apply the typeface.
Let me know if that doesn't work.

If I set subtitle to an empty string it will be inflated but I then need to set it back to null or it'll be shown (i.e. shifting my title up to display nothing). The onGlobalLayout doesn't get called until after onResume of the activity so I'm still looking into whether I can set it back in my fragment onResume or something. I'll let you know if that works out

Ahh yes, I forgot it does that. The global layout listener is called after the layout pass has finished. (onResume is approximately where its called as that is where the view is shown back to the user).
Of course your alternative is to use Toolbar and put titles inside that.

I'll probably move to use a Toolbar in the future but have a work around currently where the subtitle is set to an empty string initially and in the onGlobalLayout it'll set it to null if the subtitle is empty after setting up the view. Seems to be working well and I don't hate it as a temp fix until I get setup the toolbar everywhere.

OK thanks. Yeah I wish there was a more elegant way to handle it. Wonder if there is a default way to build in your work around

There is a new way of doing this in 2.1.1-SNAPSHOT, can you try and let me know.


possible false positive E303

Consider following valid code, at module top level:
# snip
class Foo:
    pass

class Bar:
    pass
# snip

Consider making it conditional, at module top level:
# snip
if something:
    class Foo:
        pass

    class Bar:
        pass
# snip

pep8 reports E303 too many blank lines (2) before Bar in the conditional case.
semantically it is a false positive.
syntactically it is an unclear case.
from PEP 8:
Separate top-level function and class definitions with two blank lines.
Method definitions inside a class are separated by a single blank line.
Extra blank lines blah blah blah...
this particular case doesn't fall in either clearly defined category.

I inherited a bunch of conditional classes, which was a OS-dependent implementation as suggested by garyd203 , and were basically this recipe:
http://stackoverflow.com/questions/878972/windows-cmd-encoding-change-causes-python-crash/3259271#3259271
The fix was not to worry about pep8's limitations, but completely overhaul it to move the classes to the top-level.
https://gerrit.wikimedia.org/r/#/c/239540/3/pywikibot/userinterfaces/win32_unicode.py,cm

Closing as I feel that since it doesn't fall into either of the "two line separation" cases, this should be handled with a single line of indentation between the classes. If there is a strong opinion, I would recommend reaching out to the python-dev list to try to get the PEP-8 document updated to clarify this case.


JSONPath expressions don't support extensions

Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/361
Originally created by @edgarrmondragon on 2022-04-12 22:43:42

Summary
JSONPath expressions for records and pagination tokens (see source) don't support JSONPath extensions such as filters, sorting, etc.
Steps to reproduce
Try using an expression with a filter, such as $.link[?(@.relation=='next')].url and the tap execution will fail.
What is the current bug behavior?
JSONPath extensions are not supported.
What is the expected correct behavior?
JSONPath extensions are a useful feature and should be supported.
Relevant logs and/or screenshots
NA
Possible fixes
According to the jsonpath-ng docs, extensions are supported using the extended parser:
>>> from jsonpath_ng.ext import parse

# A robust parser, not just a regex. (Makes powerful extensions possible; see below)
>>> jsonpath_expr = parse('foo[*].baz')

As a first step, a test case should be added here: https://gitlab.com/meltano/sdk/-/blob/241b38e8b6cdfa6f3397d9c70e7b9082b4594554/tests/core/test_streams.py#L156-225

View 3 previous comments from the original issue on GitLab


Update patient's UI mockup

Summary
Update patient's UI mockup

Remove edit and delete buttons
Remove images of people's faces
Change "Search" to "Input" in search bar

Checks

[ ] Link PR to issue
[ ] Add Reviewer
[ ] Pass CI

Codecov Report

Merging #70 (78bc40c) into master (70d63ef) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master      #70      +/-   ##
============================================
+ Coverage     72.15%   72.18%   +0.02%
  Complexity      399      399
============================================
  Files            70       70
  Lines          1232     1233       +1
  Branches        125      125
============================================
+ Hits            889      890       +1
  Misses          311      311
  Partials         32       32

Impacted Files
Coverage Δ

...in/java/seedu/address/commons/core/LogsCenter.java

...ss/logic/commands/exceptions/CommandException.java

...ain/java/seedu/address/logic/commands/Command.java

...eedu/address/logic/parser/DeleteCommandParser.java

.../seedu/address/storage/JsonAddressBookStorage.java

src/main/java/seedu/address/Main.java

...c/main/java/seedu/address/model/person/Person.java

src/main/java/seedu/address/ui/CommandBox.java

...rc/main/java/seedu/address/logic/LogicManager.java

...in/java/seedu/address/commons/util/StringUtil.java

... and 130 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 70d63ef...78bc40c. Read the comment docs.


Upgrade node and ruby

I rule and have completed some work on Case Manager that's ready for review!
(brief, plain english overview of your changes here)
This pull request makes the following changes:

Makes explicit the require node version, and upgrades that version to 14
Upgrades the ruby version docker container
Upgrades some dependencies

(If there are changes to the views, please include a screenshot so we know what to look for!)
It relates to the following issue #s:

Tests the fix for #2285

For reviewer:

Bring this up locally and click around to make sure everything is gravy

Followup:

After this is merged, ensure that the sandbox deploy works and looks good.
Then upgrade the Heroku stack for sandbox
Verify again that everything looks good
Promote to prod
Upgrade the stack for each prod deployment

A couple of failures during the execution of bundle exec rake knapsack:minitest. @colinxfleming @lomky want to take a look?

Nice, only down to one failing test now:
Failure:
CallListInteractionTest::completed calls section#test_0001_should add a call to completed when a call was made within 8 hrs [/home/circleci/abortioneering/test/system/call_list_interaction_test.rb:74]:
Expected false to be truthy

I would prefer to just fix the failing test instead of downgrading everything. Dependabot is going to be hitting us up for these upgrades eventually anyway. Let me see if it's an easy fix.

Codecov Report

Merging #2299 (d875a9f) into main (1197129) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #2299   +/-   ##
=======================================
  Coverage   96.25%   96.25%
=======================================
  Files          79       79
  Lines        1655     1655
=======================================
  Hits         1593     1593
  Misses         62       62

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1197129...d875a9f. Read the comment docs.


Virtualenv 20.x is not supported

I had to downgrade to virtualenv 16.7.10 to get this to work, otherwise I get the following error when I attempt a sls package:
Serverless: [pyIndividually] Installing packagings: python testproject/lib/_requirements.py testproject/requirements.txt testproject/lib
Serverless: [pyIndividually] Traceback (most recent call last):
  File "testproject/lib/_requirements.py", line 57, in <module>
    virtualenv.main()

serverless.yml
service: testproject
frameworkVersion: "2.11.1"

package:
    individually: True
    exclude:
        - "**/*"
plugins:
    - serverless-python-individually

provider:
    name: aws
    runtime: python3.6
    region: us-west-2

functions:
    testproject:
        handler: testproject/wrap.handler
        package:
            include:
                - testproject/**
custom:
    pyIndividually:
        wrap:testproject: testproject/runner.handle_message

I made a PR to fix this error : #22


Works until compiled because of custom .new

This is probably an R* or a MoarVM bug, but the tests pass when running against the source, but fail as soon as there's a compiled mbc file available:
[9:04] sterling@Weasley perl6-xml-query#master% prove -e 'perl6' t
t/basic.t .. Default constructor for 'Query' only takes named arguments
  in method new at src/gen/m-CORE.setting:970
  in block <unit> at t/basic.t:14

t/basic.t .. Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 28/28 subtests

Test Summary Report
-------------------
t/basic.t (Wstat: 65280 Tests: 0 Failed: 0)
  Non-zero exit status: 255
  Parse errors: Bad plan.  You planned 28 tests but ran 0.
Files=1, Tests=0,  1 wallclock secs ( 0.02 usr  0.01 sys +  0.54 cusr  0.05 csys =  0.62 CPU)
Result: FAIL
[9:04] sterling@Weasley perl6-xml-query#master% prove -e 'perl6 -Ilib' t
t/basic.t .. ok
All tests successful.
Files=1, Tests=28,  1 wallclock secs ( 0.02 usr  0.01 sys +  1.20 cusr  0.07 csys =  1.30 CPU)
Result: PASS

So, here you can see it passes if I make it use the local lib, but if it uses the lib from the already installed copy of the same module.
Is there a work around in the meantime aside from manually installing without letting Panda compile the mbc target?

I know it's a year later, and I'm sorry I never noticed this bug report until now, but this does not appear to happen anymore, so I'm closing this issue.


gene decreases chemical rules and data

Starting from https://github.com/NCATSTranslator/Feedback/issues/879
The rule is
(chemical a)<- decreases - (gene g) -decreases-> (chemical b) -treats-> (disease d)
The rule is coming from our mining.   But it does produce odd results like the one listed above.
Some of the problem here seems to be that TMKP has a lot of its edges reversed.  So if a paper says "chemical decreases gene' then it gets into TMKP and robokop as "gene decreases chemical". To what effect this causes the rule to exist, I am unsure, but it is in there.   It may not be the relevant feature, as there are other similar rules that are for sure driven by CTD.
Interestingly, we don't have a rule going the other direction, which I think would be much more easily supportable from a meaning standpoint if not from the statistics...

One option is to just modify the rule to be what we think it should be by reversing the directions of the chemical/gene edges.  It might be that if the TMKP edges were in the data correctly that we would have found that rule.

We want to try removing TMKP.  This will involve adding TMKP to the deny list got there edges...

@cbizon here's an example query with the deny_list:
{
  "message": {
        "query_graph": {
            "edges": {
                "e0": {
                    "predicates": [
                        "biolink:treats"
                    ],
                    "subject": "n0",
                    "object": "n1",
                    "provided_by": {
                        "denylist": [
                            "infores:tmkp"
                        ]
                    }
                }
            },
            "nodes": {
                "n0": {
                    "ids": [
                        "MONDO:0004979"
                    ],
                    "is_set": false
                },
                "n1": {
                    "categories": [
                        "biolink:SmallMolecule"
                    ],
                    "is_set": false
                }
            }
        }
    }
}


Update Installation.rst

Please update the tutorial to the latest stable Flow version to prevent confusion

Thanks for your (first!) contribution!
Unfortunately all changes go into the neos/flow-development-collection repository, this one here is a read-only split. It would be great if you could create a new PR there. Thank you!