Link to published OWASP cheatsheets rather than GitHub copies

Links to the OWASP cheatsheets should be to the published site (https://cheatsheetseries.owasp.org) rather than to the backend GitHub repo or the wiki (which has legacy versions).

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: rbsec sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


List of enrolled courses

Student should be able to see all courses in which he/she is enrolled in the dashboard page

@ShashankP19 Please review and close this. :)


Fix uni035C and uni0361

uni035C and uni0361 should be under or over the previous and the
following base glyphs.
This restores the outlines position from a previous version
There’s also a commit that removes building OTFs. There weren’t any OTFs in the original set.

Fixes #19


Add optional teardown to scenario

In JupyterLab, some scenarios involves creating some assets on the server before running the monitored code. It will be good to be able to remove those assets to have real identical iterations.
So this PR adds an optional teardown function to scenario.

Awesome, some minor lint issues, but overall looks amazing. Thank you for the additional test and documentation.


Question regarding example model and its config

Thank you for providing the PyTorch version of the Energy Transformer originally implemented in JAX. This is really helpful for me, but I have not yet been able to reproduce the results like example1.png or images in visuals directory (gif files).
Let me confirm that the example model model.pth is trained with the config file model_config.txt in the same directory, not with the config file of the same name in the parent directory energy-transformer-torch/model_config.txt? If this is correct, some parameters ('alpha': 5.0, 'attn_beta': 16.0, 'epochs': 10000) are much bigger than the hyperparameter settings reported in the paper p.16 of 2302.07253, and some ('b1': 0.99, 'b2': 0.999, 'batch_size': 512, 'learning_rate': 5e-05, 'weight_decay': 0.001) are still somewhat different.
I know that the best hyperparameters would be different since the paper uses ImageNet-1k to train the model and here uses CIFAR-10/100. I just would like to know in what configuration the example model is trained and how I can reproduce the results of example1.png or images in visuals directory.
Thanks for any additional details you can provide, I will be trying to reproduce the results in this repo.

Thanks for your reply.

it was a random model that I trained from scratch with some random chosen hyper parameters.

So, you mean that you conducted a training with randomly chosen hyperparameters without any tuning? If so, example1.png and the gif files in visuals look surprisingly well restored. It would be highly appreciated if you could find the exact configurations and let me know that, or is it really example_model/model_config.txt?

In regards to the generation of GIFs, I updated the notebook, I forgot to include interpolation=transforms.InterpolationMode.NEAREST in the Resize function.

Thanks! I will try it again.

Hey there, I have updated the code base to use accelerator from Huggingface and fix some minor errors (including plotting) from the previous code base. Masking is now randomize across batches as well. I am currently searching for a new model configuration that will be decent enough for visualization. I will give another update soon, hopefully :). It is surprising that the reconstructions were very good in the previous code base, but I don't think it will be the same for this one.


Amazing YAML update, first try

New feature - now we work with YAML files!  ^____^

According to the teacher's assignment, it is better to do this directly through the console Git CMD.
So rejected.


fix -typecheck -verify and experimental features

When using -enable-experimental-feature on a non-asserts build, we only emit an error diagnostic that has no source-line information and continue to enable the feature.
That doesn't actually prevent use of the experimental feature when you are passing -typecheck -verify, since in diagnostics verification mode, a diagnostic with an unknown error location is ignored. Thus, the experimental feature is enabled and run for type-checking, but the compiler would exit with a zero error code.
This patch takes a hammer to that escape-hatch, forcing an early non-zero exit the moment an experimental feature is requested. The error message is output to stderr so that CI and other tools should see what happened.

@swift-ci please test


Dynamically inserted importmap doesn't work in development mode

Describe the bug
Inserting an importmap at runtime via a script statement in head does not work in development mode because the script that is inserting it is placed after the module import of @vite/client. This results in the following error:
An import map is added after module script load was triggered.

This issue only affects the development mode because it it caused by the presence of the @vite/client import.
Related issues:

https://github.com/vitejs/vite/issues/9334 - the fix for this solved the issue by sorting the static importmap before the module scripts. It however doesn't take into account that classic scripts can dynamically insert importmaps.
https://github.com/vitejs/vite/issues/11304
https://github.com/vitejs/vite/pull/11492
https://github.com/vitejs/vite/issues/11304

Workaround: trick Vite into thinking that the script that is inserting the importmap is an importmap itself - this will make sure that it gets sorted before the module import of @vite/client. This can be achieved by adding an attribute to it that matches this regular expression (e.g. data-vite-workaround="type=importmap").
Reproduction
https://stackblitz.com/edit/vitejs-vite-yde4az?file=index.html
Steps to reproduce
Run npm install && npm run dev.
Expected result:

The page shows the text "Hello World!"

Actual result:

The page shows the text "Waiting for script..."

Note that running npm install && npm run build && npm run preview works as expected.
System Info
Output from running the command in the linked StackBlitz:

System:
    OS: Linux 5.0 undefined
    CPU: (8) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Memory: 0 Bytes / 0 Bytes
    Shell: 1.0 - /bin/jsh
  Binaries:
    Node: 18.18.0 - /usr/local/bin/node
    Yarn: 1.22.19 - /usr/local/bin/yarn
    npm: 9.4.2 - /usr/local/bin/npm
    pnpm: 8.10.5 - /usr/local/bin/pnpm
  npmPackages:
    vite: ^5.0.2 => 5.0.4

Used Package Manager
npm
Logs
No response
Validations

[X] Follow our Code of Conduct
[X] Read the Contributing Guidelines.
[X] Read the docs.
[X] Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
[X] Make sure this is a Vite issue and not a framework-specific issue. For example, if it's a Vue SFC related bug, it should likely be reported to vuejs/core instead.
[X] Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
[X] The provided reproduction is a minimal reproducible example of the bug.

@sorin-davidoi  Would you be able to provide some more information about how to work around this?

I faced the same issue. Worked around it now by combining a build with watch mode + preview to simulate "dev" mode instead of using the "normal" vite dev-server.
One way I can think of, to solve this, would be to add a new option in the vite config file to disable the insertion of the @vite/client script. When someone would disable it, they still have the freedom to manually load the @vite/client from javascript in case they want to have the auto-reload / hmr functionality during dev mode. E.g. they could add something like this in their code (after the importmap has been added):
if (import.meta.env.MODE === 'development') {
  const origin = new URL(import.meta.url).origin;
  import(/* @vite-ignore */ `${origin}/@vite/client`);
}

I have been able to solve this using a the api plugin transformindexhtml.
https://vitejs.dev/guide/api-plugin#transformindexhtml
Example.
const htmlPlugin = () => {
    return {
        name: 'html-transform',
        transformIndexHtml(html) {
            const newHtml = html.replace(
                /<script type="module" src="\/@vite\/client"><\/script>/,
                `<script src="./importmap.js"></script><script type="module" src="/@vite/client"></script>`,
            )
            return newHtml
        },
    }
}

const developmentConfig = () => ({
        server: {
            hmr: true
        },
        plugins: [
            htmlPlugin()
        ]
});


Android Client not properly published in Google Play

Hello,
I think the mobile-phones integration should be very insteresting for the evolution of this project.
I recently look for android the client app without lucky.
https://play.google.com/store/search?q=keybase
I can see a andoird client in your code, could you publish the client for andoird?
Marc

I think the Android client is currently in testing-only mode, so doesn't appear in the main/public Play store results yet

Is it possible to get access to help test out the mobile app?

@dwallraff I got access by responding to this request but I don't know if they are still needing testers

Cool. Thanks @stevejalim!


Update particular series color on time interval does not work.

Here is the code to update color of a particular series in highcharts, it works when i move the mouse over the legend. it has to get update automatically without any event.
http://jsfiddle.net/rick822/k7pbwch4/

I'm not sure what issue you are facing.


I don't see the widget in my list of widgets...

No idea if I'm doing something wrong, but I haven't found a way to add the widget to my home screen. (And I would really like to do so!) I just don't see it in the widgets menu, either by scrolling or searching.
I've tried restarting my phone, and that didn't help.
I'm all up to date on iOS (16.3.1).
Any ideas?

Hi,
Are you using the workflow described here? https://support.apple.com/en-us/HT207122
Here's what it looks like on my device:

Sorry to bother you! There must be some kind of delay between install and showing up in that widgets list, because it was there when I checked again just now.

Cool project, btw!!! ❤ ❤ ❤

Glad to hear it!

Thanks! Curious how you found out about this project?

That same stack overflow answer on scroll views. 😂 (I looked at the other closed issues, and saw I'm not the only one! heh.) Of note, the link in your comment is broken, though I think the one in the answer is correct. I ended up using PDFKit's PDFView, but if I end up needing to customize it, I may be back to look at your view. (Though the app I'm working on is primarily is UIKit, it's got one or two areas I'm "borrowing" SwiftUI views. Heh.)

Yep, unfortunately can't edit comments :(


Bug 2104480: remove namespace from rules request

Signed-off-by: Matan Schatzman<EMAIL_ADDRESS>
📝 Description
on production env, request to promethous /rules come back empty when there is namespace in the request.
🎥 Demo

Please add a video or an image of the behavior/changes

/lgtm


Fix translations by reconfiguring on update #158341090

Overview
Reconfigure on update.
This will cause translations in config.xml to be regenerated whenever we deploy.
Checklist

[x] PR has a name that won't get you publicly shamed for vagueness
[ ] Files changed in the PR have been yapf-ed for style violations

Testing Instructions
First let's get to the state where staging is at, then we'll fix it.

Turn off vagrant rsync-auto if running
Check out the commit prior to the one where translations were fixed (bebecba)
vagrant reload to make the changes to not exclude the xmlconfig.po files take effect
./script/server
Visit http://localhost:8080/districtmapping/plan/2/view/ and notice that the translations in the metrics pane to the right are broken
Check out this branch
vagrant reload again (xmlconfig.po is now excluded and won't overwrite what's in the VM/container)
Run ./scripts/server and make sure the translations are still broken

Now we are in the state staging is at. Let's fix it by running:

./scripts/update
./scripts/server

These are the same commands that are run when we deploy (see https://github.com/azavea/district-builder-dtl-pa/blob/develop/scripts/infra#L86-L87) so this should fix it.
Closes #158341090

So I don't think this perfectly handles the edge case where the .po files have disappeared completely, but it does seem to work if a .po file exists but is outdated. I think that's the case we're going to encounter during deployment.

That's correct. I think we'll always have some .po files in the container image because they're checked into source code, so I don't think we need to worry about that first edge case during deployments.


Verify .next target for all layout constraints

Not sure if next sibling is correctly passed from the renderer/reconciler to auto layout extensions in TokamakUIKit, this needs to be verified and fixed if broken. Good to have some demo screens in the Example project for verification.

Closing as outdated


A few quick fixes to try to reduce max memory usage in the tests

This helps with #4315 and #4542
I didn't take the time to measure the actual maximum memory usage by these tests. I plan on making some changes to do that soon.  For now this just does the most obvious things. It disables the test for #4315 as it is not critical. It also splits up the test for #4542 into smaller tests to try to reduce the total memory usage.

build


[SPARK-48793][SQL][TESTS] Unify v1 and v2 ALTER TABLE .. DROP|RENAME COLUMN[S] ... tests

What changes were proposed in this pull request?
The pr aims to:

Move parser tests from o.a.s.s.c.p.DDLParserSuite and o.a.s.s.c.p.ErrorParserSuite to AlterTableRenameColumnParserSuite & AlterTableDropColumnParserSuite
Add a test for DSv2 ALTER TABLE .. DROP|RENAME to v2.AlterTableDropColumnSuite & v2.AlterTableRenameColumnSuite

(This PR includes the unification of two commands: DROP COLUMN & RENAME COLUMN)
Why are the changes needed?

To improve test coverage.
Align with other similar tests, eg: AlterTableRename*

Does this PR introduce any user-facing change?
No, only tests.
How was this patch tested?

Add new UT
Pass GA.

Was this patch authored or co-authored using generative AI tooling?
No.

cc @cloud-fan

thanks, merging to master!


Docker setup fails with permissions error

Summary
After a fresh install, attempted to run arthur:
./bin/run_arthur.sh
++ pwd
+ docker run --rm --interactive --tty {volumes omitted} --env DATA_WAREHOUSE_CONFIG=/opt/data-warehouse/warehouse_config --env ARTHUR_DEFAULT_PREFIX=bhtucker arthur-redshift-etl:latest
+ cd /opt/src/arthur-redshift-etl
+ python3 setup.py --quiet develop
error: could not create 'python/redshift_etl.egg-info': Permission denied

Details
Prior steps were only git clone and ./bin/build_arthur.sh
Propose label: Bug, maybe documentation bug?

Here's what I ran:
docker image rm arthur-redshift-etl
docker system prune
rm -rf arthur-redshift-etl
git clone git@github.com:harrystech/arthur-redshift-etl.git
cd arthur-redshift-etl
bin/build_arthur.sh
bin/run_arthur.sh

This sequence of commands puts me into a Docker image running Arthur.
Here's the state of next:
6265949 (HEAD -> next, origin/next, origin/HEAD) Merge branch 'master' into next
9e1568c Merge pull request #236 from harrystech/flake8-fixes
d32b550 (tag: v1.28.0, origin/master) Merge remote-tracking branch 'origin/next'

(The additional commit on next just changed some comments, not related to our Docker setup.)
Please be sure to be on the latest version as shown above. "permission denied" may have popped up during development on next when the user arthur inside the Docker container wasn't the owner of /opt/src.
To debug, please re-run the docker command and add: --entrypoint bash right before the image tag.
In the shell, please take a look at:
ls -la /opt/src/arthur-redshift-etl/
cd /opt/src/arthur-redshift-etl/
touch hello
python setup.py develop

and let me know what error messages show up.

Will do, thanks @tvogels01 !

Then, the run_arthur.sh test:
./bin/run_arthur.sh
You must set DATA_WAREHOUSE_CONFIG when not specifying the config directory.

Ok, fair enough, I'll set one (to an existing directory on my machine):
export DATA_WAREHOUSE_CONFIG=/home/bhtucker/third_party/warehouse_config/
$ ./bin/run_arthur.sh
++ pwd
+ docker run --rm --interactive --tty --volume /home/bhtucker/third_party:/opt/data-warehouse --volume /home/bhtucker/third_party/arthur-redshift-etl:/opt/src/arthur-redshift-etl --volume /home/bhtucker/.aws:/home/arthur/.aws --volume /home/bhtucker/.ssh:/home/arthur/.ssh:ro --env DATA_WAREHOUSE_CONFIG=/opt/data-warehouse/warehouse_config --env ARTHUR_DEFAULT_PREFIX=bhtucker arthur-redshift-etl:latest
+ cd /opt/src/arthur-redshift-etl
+ python3 setup.py --quiet develop
error: [Errno 13] Permission denied

With 'probes':
docker run --rm --interactive --tty --volume /home/bhtucker/third_party:/opt/data-warehouse --volume /home/bhtucker/third_party/arthur-redshift-etl:/opt/src/arthur-redshift-etl --volume /home/bhtucker/.aws:/home/arthur/.aws --volume /home/bhtucker/.ssh:/home/arthur/.ssh:ro --env DATA_WAREHOUSE_CONFIG=/opt/data-warehouse/warehouse_config --env ARTHUR_DEFAULT_PREFIX=bhtucker --entrypoint bash arthur-redshift-etl:latest
(venv) (aws:, prefix:bhtucker) $ ls -la /opt/src/arthur-redshift-etl/
total 516
drwxrwxr-x 15   1002   1005   4096 Aug 28 16:44 .
drwxr-xr-x  1 arthur arthur   4096 Aug 28 16:45 ..
drwxrwxr-x  4   1002   1005   4096 Aug 27 22:14 .arthurenv
-rw-rw-r--  1   1002   1005    430 Aug 27 22:06 .dockerignore
-rw-rw-r--  1   1002   1005    459 Aug 27 22:06 .editorconfig
drwxrwxr-x  8   1002   1005   4096 Aug 28 16:44 .git
drwxrwxr-x  3   1002   1005   4096 Aug 27 22:06 .github
-rw-rw-r--  1   1002   1005    406 Aug 27 22:06 .gitignore
-rw-rw-r--  1   1002   1005   2728 Aug 27 22:06 Dockerfile
-rw-rw-r--  1   1002   1005   5731 Aug 27 22:06 INSTALL.md
-rw-rw-r--  1   1002   1005   1070 Aug 27 22:06 LICENSE
-rw-rw-r--  1   1002   1005  19208 Aug 27 22:06 README.md
-rw-rw-r--  1   1002   1005    440 Aug 27 22:06 TODO.md
-rw-rw-r--  1   1002   1005 381244 Aug 27 22:51 arthur.log
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:06 bin
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:06 cloudformation
drwxrwxr-x  2   1002   1005   4096 Aug 27 23:07 dist
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:06 etc
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:06 githooks
drwxrwxr-x  3   1002   1005   4096 Aug 27 22:06 log_processing
drwxrwxr-x  5   1002   1005   4096 Aug 27 22:15 python
-rw-rw-r--  1   1002   1005   2469 Aug 27 22:06 readme_release.md
-rw-rw-r--  1   1002   1005    149 Aug 27 22:06 requirements-dev.txt
-rw-rw-r--  1   1002   1005    131 Aug 27 22:06 requirements-linters.txt
-rw-rw-r--  1   1002   1005    218 Aug 27 22:06 requirements.txt
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:52 schemas
-rw-rw-r--  1   1002   1005   1543 Aug 28 16:44 setup.cfg
-rw-rw-r--  1   1002   1005   1565 Aug 27 22:06 setup.py
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:06 sql
drwxrwxr-x  2   1002   1005   4096 Aug 27 22:06 wiki
(venv) (aws:, prefix:bhtucker) $ cd /opt/src/arthur-redshift-etl/
(venv) (aws:, prefix:bhtucker) $ touch hello
touch: cannot touch 'hello': Permission denied
(venv) (aws:, prefix:bhtucker) $ python setup.py develop
running develop
running egg_info
error: [Errno 13] Permission denied

So I suppose arthur the container user isn't allowed to talk back out to my src dir. I confess I don't use volumes for anything but read-only config files so don't know how this should work.

I'll have to dig in to Docker volumes to see what might cause this issue. For me, the owner of /opt/src stays arthur.
To unblock you, I'd suggest that you switch into your warehouse directory before starting arthur. The reason is that this will put the image into "standalone" mode which means it won't try to run python setup.py develop.
Here's what happens on my laptop:
~/repos/harrystech/arthur-redshift-etl/bin/run_arthur.sh
Did not find source path (looked for setup.py) -- switching to standalone mode.
Changes to code in /opt/src/arthur-redshift-etl will not be preservd between runs.
However, changes to your schemas or config will be reflected in your local filesystem.
+ docker run --rm --interactive --tty ... --env DATA_WAREHOUSE_CONFIG=/opt/data-warehouse/config_data_development --env ARTHUR_DEFAULT_PREFIX=tom ... arthur-redshift-etl:latest

Running from the 'warehouse' directory and not from arthur source directory seems wise.
I also forgot the sibling setup is 'repo' then 'config' adjacent to e.g. 'sources'.  Added that layer.
In fact I did a pip install -e . before running in Docker (muscle memory). So I cleared it out and tried again without that; the error simply changes slightly to error: could not create 'python/redshift_etl.egg-info': Permission denied.
Perhaps I have some global docker settings or version info I don't know about. Does classic the virtualenv setup still work? That's my preference anyway :)

Using a virtual env might work but I haven't tested in a while. I don't see why it wouldn't.
Here's what the permissions should look like:
$ ls -lad /opt/ /opt/data-warehouse/ /opt/local/ /opt/src/ /opt/src/arthur-redshift-etl/python/redshift_etl.egg-info/
drwxr-xr-x  1 root   root   4096 Aug 28 11:40 /opt/
drwxr-xr-x 37 arthur arthur 1184 Aug 28 16:55 /opt/data-warehouse/
drwxr-xr-x  1 arthur arthur 4096 Aug 28 11:40 /opt/local/
drwxr-xr-x  1 arthur arthur 4096 Aug 28 11:40 /opt/src/
drwxr-xr-x  8 arthur arthur  256 Aug 28 17:32 /opt/src/arthur-redshift-etl/python/redshift_etl.egg-info/

Note how the user stays arthur.
In the end, this is the directory structure that you're aiming for:
top/warehouse/config/
top/warehouse/schemas/
top/arthur-redshift-etl/bin/
top/arthur-redshift-etl/etc/
top/arthur-redshift-etl/python/
...

For now, what happens if you simply comment out the line python setup.py develop in bin/entrypoint.sh?
It's not needed unless you develop code inside the Docker container. The other lines (setting PATH and activating the virtual env) are needed.

Please take try out: Allow read-only source directory #242
When the Docker image is created, an install step already installs the ETL source. So the python setup.py develop is great but not necessary to have a running container.


Adding fedora in map.jinga

When I tried to use nfs.mount in Fedora, I had an error. It seems package list is missing in map.jinga. I just have copied and pasted RedHat conf

@mioux did #25 fix your problem?

Hello
Nope, i have an error with all my minions. One is running debian (and had no problem) the other is running fedora.
I'll post the error message when I can access a computer, I have checked on my phone.

Here I have joined all error logs.
I have the same issue with both VM now 😄
error-debian(server).log
error-fedora(client).log
FIY : the states files includes 'nfs.mount' for the fedora, and 'nfs.server' for the debian

@mioux fixed it in #26

Seems to be OK now :smile_cat:
Thanks


Using LIFXKit in iOS App extensions

Issue :
The latest version of LIFXKit (0.6.1) won't compile in an iOS App extension, using Xcode 6.2 and iOS 8.2 SDK.
Cause :
Usage of +[UIApplication sharedApplication] in :

LFXWiFiObserverIOS.m, line 30
LFXLocalTransportManager.m, line 86

sharedApplication is declared as NS_EXTENSION_UNAVAILABLE_IOS("Use view controller based solutions where appropriate instead."); in UIApplication.h line 215.
Possible solution :
Make the user define a preprocessor macro (such as #LIFXKIT_APP_EXTENSIONS) to disable codes that calls unavailable APIs when compiling an app extension. That's what AFNetworking does (see issue #2119.

In the meantime, could you confirm that commenting the two blocks of codes using sharedApplication when compiling an app extension (Today widget / Apple watch app) won't have any incidence ?
I'm pretty sure it won't, but you know the code better than me ahah !
Thanks,
Max.

Have the same issue here. While for AFNetworking, they just use it to show the network activity indicator, here they use the applicationState to determine if they still need to process some information (I guess?).
An answer  from the developers would be great on this.

@depl0y: I havn't noticed any issue in a watch app commenting these lines. There might be some underlying issues though.


NTR: spore-bearing structure development

Fungal terms currently under "spore-bearing organ development" will be moved under this new term "spore-bearing structure development" which is is technically more accurate for fungi. The synonym is "sporophore." The scientific term "sporophore" generically refers to the spore-bearing structures of any fungal species...conidiophore, basidiocarp, ascocarp, sporangiophore, cleistothecium, perithecium, zygosporangium, etc. "Sporangium" is used for spore-bearing structure (spore sac, spore case) in both plants and fungi.
Are there guidelines for when to use "formation" versus "development?" I could see also this term as "spore-bearing structure formation." There are both formats in the lineage.
GO:0048856 anatomical structure development
GO:0048646 anatomical structure formation involved in morphogenesis
For conidia and conidiophores, the choice of formation vs. development affects the term lineage. The coniophore is composed of several different cell types, including metulae and phialides and ultimately conidia. I don't see why metula and phialide are not "formation" terms or "conidium development" is not a distinct term. Existing terms:
-conidiophore development (structure made of many cells of different cell-types)
-- metula development (a cell-type that forms part of a conidiophore)
---phialide development (a cell-type that forms part of a conidiophore)
---conidiophore stalk development (a cell-type that forms part of a conidiophore)
---conidium formation (a cell-type produced by a conidiophore)
My thought is the term development requires the structure or cell-type to at least exist in some form. An impact on development might be the failure to reach its fully mature state (spore cell walls are defective or a cell-type is missing). Examples:
conidium formation...the ability to generate conidia (existing term)
conidium development...the ability to generate a fully mature, properly assembled, viable conidium. (not a current term but perhaps should be).
I am sifting through fungal terms and one issue is that "development" is used interchangeably for actual developmental processes and in place of "formation/generation/production" and this brings in more terms that cause issues in definitions. "Differentiation" is a parent of "cell development," but the specific issues will have their own tickets.

@paolaroncaglia, going to punt this to you as part of the spore-bearing organ clean-up.#12578

@dianeoinglis  - can you create 'spore-bearing structure' now in FAO?

Note for self - see http://wiki.geneontology.org/index.php/Ontology_meeting_2016-08-25#FAO_update

Hi Tanya, Has this term been created to use a GOID as xref?
https://github.com/obophenotype/fungal-anatomy-ontology/issues/2

Setting to pending while we wait for @dianeoinglis to create 'spore-bearing structure' in FAO.

A term with the synonym "spore-bearing organ" already exists but needs to modified to match the GO term change. The GO term should probably get the synonym "sporophore" which is the primary term name for FAO:0000010. I am having an uber-admin password issue with installation of JavaSE 6 that seems to be required before OBO-edit can be installed. I
FAO:0000010
name: sporophore
def: "A structure directly within which spores are contained or from which spores are formed." [FAO:curators]
xref: GO:0075259
synonym: "spore-bearing organ" RELATED [] ADD:-synonym: "spore-bearing structure" RELATED []
is_a: FAO:0000001 ! fungal structure

For each fungal development term, a "formation" term should also be created. The development term refers to the formation of a normal, fully mature structure and requires the presence of the structure in some form. The formation term is for the ability to produce the structure at all..
"spore-bearing structure formation"
def: The  process of generating a spore-bearing organ. A spore-bearing organ is an anatomical structure that produces new spores.

Note for self: also see http://wiki.geneontology.org/index.php/Ontology_meeting_2016-09-22#FAO_update and https://github.com/obophenotype/cell-ontology/issues/447

Hi @paolaroncaglia
Note, this term already exists as a synonym to FAO:0000010 'sporophore'
See https://github.com/obophenotype/fungal-anatomy-ontology/issues/11
The FAO.obo has not yet been updated but all new terms listed in the FAO issue tracker are being assigned the next available ID and the FAO will modified in the near future after I complete the addition of more modifications and terms. Thanks!

Hi @dianeoinglis,
Wrapping up this ticket. I will:

[ ] rename ”spore-bearing organ development" as “spore-bearing structure development”; revise its def to replace ‘organ’ with ‘structure’; add exact synonym "sporophore development”, and broad synonym “sporangium development”; give it the logical definition is_a anatomical structure development, results_in_development_of FAO:0000010 sporophore (it doesn’t matter if FAO:0000010 will be renamed “spore-bearing structure”)

[ ] add the new term ”spore-bearing structure formation"
is_a developmental process
part_of spore-bearing structure development [according to the guidelines on development (and formation) terms here http://geneontology.org/page/development]
def: The process of generating a spore-bearing structure. A spore-bearing structure is an anatomical structure that produces new spores.
dbxref: GOC:di
exact synonym: sporophore formation
broad synonym: sporangium formation

Please comment here if you have any concern with the above. Otherwise, I will proceed after the holidays.
If you require additional, more specific ‘formation’ or ‘development’ terms for fungal structures, please open a new ticket, and include PubMed references etc. as outlined in https://github.com/geneontology/go-ontology/blob/master/CONTRIBUTING.md
Many thanks!


Using the classifier

Hi
After saving the model in news-category-classification, how do you actually use it to predict text classification?
Can you put up an example, please?

@davidniki02
I wrote simple sample code to know how to save/load the model.
After loading the model, you can predict/do classification as you did.
# save model architecture
model_json = model.to_json()
with open("model.json", "w") as json_file :
    json_file.write(model_json)

# save weights
model.save_weights("model.h5")

# load model architecture
from keras.models import model_from_json
with open("model.json", "r") as json_file:
    loaded_model_json = json_file.read()
loaded_model = model_from_json(loaded_model_json)

# load weights
loaded_model.load_weights("model.h5")

@lyeoni thanks, but I think we also need to store the tokenizer? I stored it using pickle.
Here is my code but it predicts the same category all the time:

from keras.models import load_model
import keras.preprocessing.text as kpt
import numpy as np
import pandas as pd
import pickle

# loading
model = load_model('model_20190519141343.h5')

handle = open('tokenizer.pickle', 'rb')
tk = pickle.load(handle)

data = pd.read_json('News_Category_Dataset_v2.json', lines=True).drop(['authors', 'date', 'link'], axis=1)

while 1:
	text = input("Say something: ")

	if len(text) == 0:
		break

	#tk.fit_on_texts(text)
	#converts the texts to the index equivalents in our dictionary
	pred = tk.texts_to_sequences([text])
	print(pred)

	#onehot representation of all words in the evaluation text, and how they appear in our dictionary
	#input = tk.sequences_to_matrix(pred, mode='binary')

	arr = np.zeros(50).reshape(1, 50)
	print(arr)
	for i, word in enumerate(pred[0]):
		arr[0][i] = word
	print(arr)

	prediction = model.predict(arr)
	print(prediction)
	cls = np.argmax(prediction)

	print(cls)
	print(data['category'][cls])

@davidniki02
It depends on what kind of tokenizer you use.
For example, if you use nltk.mosestokenizer (in nltk.tokenize.moses), you don't need to save/load the saved tokenizer. Just call the function, and use returned tokenizer instance.
>>> m = MosesTokenizer()
>>> m.tokenize('2016, pp.')
    [u'2016', u',', u'pp', u'.']

Thanks @lyeoni,
You are using MosesTokenizer in tokenization_en.py but in data_loader.py, it is using Tokenizer from keras.preprocessing.text
I updated token_to_index in data_loader to store the tokenizer:
self.tokenizer = tokenizer
then saved it during training:
    # save model
    model.save('model_'+current+'.h5')
    print('MODEL SAVED')

	# saving
    with open('tokenizer.pickle', 'wb') as handle:
        pickle.dump(loader.tokenizer, handle, protocol=pickle.HIGHEST_PROTOCOL)

The problem is it predicts the same category all the time (with the same Huffpost dataset you have used)
Also, MosesTokenizer does not have any texts_to_sequences so I don't think it can be used directly for prediction?

Any luck, @lyeoni ?

Sorry for the delay in replying,
The reason why I used 2 tokenizers(MosesTokenizer,  Keras tokenizer)  is:

The actual tokenization is only preformed by MosesTokenizer in tokenization.py. Keras tokenizer in data_loader.py is only used to easily change words into indexes and build vocabulary.
So, if you tokenize corpus (already tokenzied by MosesTokenizer) using Keras tokenizer , there's not much difference.

@davidniki02,
I want to know if your tokenizer is properly saved/loaded. Because I'm not sure of if the method to save tokenizer works well. Please, check that the sample text is correctly tokenized.

Thanks for replying @lyeoni
Using MosesTokenizer does not return the numeric representation of the array we need to pass to model.predict
Here is the latest not working code:
# loading
model = load_model('model_20190519141343.h5')

handle = open('tokenizer.pickle', 'rb')
#tk = pickle.load(handle)
tk = MosesTokenizer()

data = pd.read_json('News_Category_Dataset_v2.json', lines=True).drop(['authors', 'date', 'link'], axis=1)

while 1:
	text = input("Say something: ")

	if len(text) == 0:
		break

	#tk.fit_on_texts(text)
	#converts the texts to the index equivalents in our dictionary
	tokens = tk.tokenize(text.strip(), escape=False)
	print(tokens)

	arr = np.zeros(50).reshape(1, 50)
	print(arr)
	for i, word in enumerate(tokens):
		arr[0][i] = word
	print(arr)

	prediction = model.predict(arr)
	print(prediction)
	cls = np.argmax(prediction)

	print(cls)
	print(data['category'][cls])

I probably need to use keras tokenizer to convert it to numbers (and get rid of the numpy array), e.g.
tokenizer = Tokenizer(num_words = 50000+1, oov_token='UNK')
tokenizer.texts_to_sequences(tokens)

but I don't know if reload the corpus and concatenate the new text to it, if I need to fit_text etc.
# token_to_index
tokens = tokenized_corpus.apply(lambda i: i.split())
tokenizer.fit_on_texts(tokens)
tokenizer.word_index = {word:index for word, index in tokenizer.word_index.items() if index <= 50000}

It's getting a bit confusing. Can you show how the code should actually look?

@davidniki02 ,
In your code, tokenizer is initialized/fit every time.
But, fit_on_texts function should be called for all corpus that you have.
(reference link: http://faroit.com/keras-docs/1.2.2/preprocessing/text/)

fit_on_texts(texts):

Arguments:

texts: list of texts to train on.

And, you don't have to use MosesTokenizer because Keras Tokenizer work enough well.
Without MosesTokenizer, just use following code. (tokenized_corpus could be the original corpus, not pre-processed)
def token_to_index(self, tokenized_corpus, maximum_word_num):
   tokenizer = Tokenizer(num_words = maximum_word_num+1, oov_token='UNK')

   # tokenizer fitting (token to index number)
   tokens = tokenized_corpus.apply(lambda i: i.split())
   tokenizer.fit_on_texts(tokens)

   # build vocabulary
   tokenizer.word_index = {word:index for word, index in tokenizer.word_index.items() if index <= maximum_word_num}
   vocabulary = tokenizer.word_index

   # texts_to_sequences changes words into indexes
   return vocabulary, tokenizer.texts_to_sequences(tokens)

Thanks @lyeoni , that is exactly the part I don't get:
How do I use the tokenizer on a new text (e.g. passed from the command prompt)?
The code from data_loader.py loads the corpus and fits the tokenizer, and I altered the code a bit to store the tokenizer for later access: self.tokenizer = tokenizer
But using the tokenizer to transform a new text (hence the text = input("Say something: ") code) into something the model would understand is the troublesome part. Can you show a sample for that, please?

@lyeoni, I think I got it right this time:
from keras.models import load_model
from nltk.tokenize.moses import MosesTokenizer
import keras.preprocessing.text as kpt
from keras.preprocessing.sequence import pad_sequences
from keras.preprocessing.text import Tokenizer, text_to_word_sequence
import numpy as np
import pandas as pd
import pickle
import data_loader

# load model architecture
from keras.models import model_from_json

# loading
model = load_model('model_20190519141343.h5')

handle = open('tokenizer.pickle', 'rb')
tk = pickle.load(handle)

loader = data_loader.DataLoader("corpus.tk.txt", "corpus.tk.vec.txt", "News_Category_Dataset_v2.json")
loader.load_cat()
print(loader.category_dict)
while 1:
	text = input("Say something: ")

	if len(text) == 0:
		break

	seq = tk.texts_to_sequences([text.strip()])
	x = pad_sequences(seq, 50)
	print(x)
	prediction = model.predict(x)
	print(prediction)
	cls = np.argmax(prediction)

	print(cls)
	print(loader.category_dict[cls])

That being said, it seems the predictions get really off sometimes. I have trained them on the headlines which yields a higher accuracy than summaries (80%) but when tested against something like "Facebook Accused Of Reading Texts And Accessing Microphones In Lawsuit" (which is even in the News dataset) the answer is "POLITICS"
What results do you get? How accurate is the model?


FSDP not working

Please check that this issue hasn't been reported before.

[X] I searched previous Bug Reports didn't find any similar reports.

Expected Behavior
That it works without errors when running. I tested with bf16, not working. qlora also will not work.
Current behaviour
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 36, in <module>
    fire.Fire(do_cli)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 32, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/train.py", line 116, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1575, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1954, in _inner_training_loop
    self.optimizer.step()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/optimizer.py", line 145, in step
    self.optimizer.step(closure)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 69, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 280, in wrapper
    out = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 268, in step
    self.prefetch_state(p)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 320, in prefetch_state
    F.prefetch_tensor(state['state1'])
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/functional.py", line 187, in prefetch_tensor
    lib.cprefetch(get_ptr(A), ct.c_size_t(num_bytes), ct.c_int32(deviceid))
TypeError: 'NoneType' object cannot be interpreted as an integer
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 36, in <module>
    fire.Fire(do_cli)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 32, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/train.py", line 116, in train
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1575, in train
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1954, in _inner_training_loop
    self.optimizer.step()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/optimizer.py", line 145, in step
    self.optimizer.step(closure)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 69, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 280, in wrapper
    out = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 268, in step
    self.prefetch_state(p)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 320, in prefetch_state
    F.prefetch_tensor(state['state1'])
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/functional.py", line 187, in prefetch_tensor
    lib.cprefetch(get_ptr(A), ct.c_size_t(num_bytes), ct.c_int32(deviceid))
TypeError: 'NoneType' object cannot be interpreted as an integer
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 36, in <module>
    fire.Fire(do_cli)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    component = fn(*varargs, **kwargs)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 32, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/train.py", line 116, in train
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1575, in train
    exec(code, run_globals)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 36, in <module>
    fire.Fire(do_cli)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 475, in _Fire
    return inner_training_loop(
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1954, in _inner_training_loop
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.10/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/cli/train.py", line 32, in do_cli
    train(cfg=parsed_cfg, cli_args=parsed_cli_args, dataset_meta=dataset_meta)
      File "/workspace/Boost-Patent/bp-train/axolotl/axolotl/src/axolotl/train.py", line 116, in train
self.optimizer.step()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/optimizer.py", line 145, in step
    trainer.train(resume_from_checkpoint=resume_from_checkpoint)
  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1575, in train
    self.optimizer.step(closure)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 69, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 280, in wrapper
    out = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 268, in step
        return inner_training_loop(self.prefetch_state(p)

  File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 1954, in _inner_training_loop
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 320, in prefetch_state
    F.prefetch_tensor(state['state1'])
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/functional.py", line 187, in prefetch_tensor
    lib.cprefetch(get_ptr(A), ct.c_size_t(num_bytes), ct.c_int32(deviceid))
TypeError: 'NoneType' object cannot be interpreted as an integer
    self.optimizer.step()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/optimizer.py", line 145, in step
    self.optimizer.step(closure)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 69, in wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 280, in wrapper
    out = func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 268, in step
    self.prefetch_state(p)
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/optim/optimizer.py", line 320, in prefetch_state
    F.prefetch_tensor(state['state1'])
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/functional.py", line 187, in prefetch_tensor
    lib.cprefetch(get_ptr(A), ct.c_size_t(num_bytes), ct.c_int32(deviceid))
TypeError: 'NoneType' object cannot be interpreted as an integer

Steps to reproduce
Rented 4x RTX 4090 GPUs from runpod and compiled axolotl from main.
base_model: meta-llama/Llama-2-7b-hf
base_model_config: meta-llama/Llama-2-7b-hf
model_type: LlamaForCausalLM
tokenizer_type: LlamaTokenizer
is_llama_derived_model: true

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: x
    type: alpaca
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./model-out

adapter:
lora_model_dir:

sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true

wandb_mode:
wandb_project:
wandb_entity:
wandb_watch:
wandb_run_id:
wandb_log_model:

gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 3
optimizer: paged_adamw_32bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: true
bfloat16: true
fp16: false
tf32: false

gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true

warmup_steps: 10
eval_steps: 20
save_steps:
debug:
deepspeed:
weight_decay: 0.0
fsdp:
  - full_shard
  - auto_wrap
fsdp_config:
  fsdp_offload_params: true
  fsdp_state_dict_type: FULL_STATE_DICT
  fsdp_transformer_layer_cls_to_wrap: LlamaDecoderLayer
special_tokens:
  bos_token: "<s>"
  eos_token: "</s>"
  unk_token: "<unk>"

Possible solution
No response
Which Operating Systems are you using?

[X] Linux
[ ] macOS
[ ] Windows

Python Version
3.10
axolotl branch-commit
21ec195
Acknowledgements

[X] My issue title is concise, descriptive, and in title casing.
[X] I have searched the existing issues to make sure this bug has not been reported yet.
[X] I am using the latest version of axolotl.
[X] I have provided enough information for the maintainers to reproduce and diagnose the issue.

I don't believe FSDP works with lora or qlora. I tested FSDP on main and it seems to work


Active Passive ELB ILB AZ README update

README deploy button links pointed to the non Availability Zones templates.

Closing PR incorrectly included files for another branch, will fix and resubmit.

Reopening


Link to Springerle Cookie Jar

https://github.com/Springerle/cookie-jar

@jhermann If you rebase this commit, we'll accept it in. Apologies for the delay.

Not sure how to proceed on this guy. Technically https://github.com/Springerle/cookie-jar is not a cookiecutter template,  yet it would make sense to refer to it.
Should we have a new section? (That's what this PR does) :confused:

:bell:


dropdown toggles content (dropdown menus) not showing when add .show class

Bootstrap 4 work incorrectly, when add .show class to the .dropdown element, dropdown menus not showing

    Dropdown link

    Action
    Another action
    Something else here

when add .show class to the .dropdown-menu element, dropdown menus is showing

You need show on the dropdown-menu element itself:
https://github.com/twbs/bootstrap/blob/v4-dev/scss/_dropdown.scss#L147
The .dropdown is provided just to wrap with relative positioned element:
http://getbootstrap.com/docs/4.0/components/dropdowns/#examples

@peterblazejewicz
In bootstrap v3.3.7, add .open class to the .dropdown element, dropdown menus is showing
In bootstrap v4.0.0-alpha.6, add .show class to the .dropdown element, dropdown menus is showing
https://v4-alpha.getbootstrap.com/components/dropdowns/
But, in bootstrap 4, add .show class to the .dropdown element, dropdown menus not showing
https://getbootstrap.com/docs/4.0/components/dropdowns/#single-button-dropdowns
Dropdown menus is showing when add .show class to the .dropdown-menu element, because it change css code from
.show > .dropdown-menu {
  display: block;
}

to
.dropdown-menu.show {
  display: block;
}

I think this feature of previous version very  good
I don't know bootstrap try to change the css code or missing css code

Dropdowns in v4 are working as intended and there are no planned changes.


Managing Role Limits in Seedfarmer Environments

Hello,
We are using Seedfarmer extensively across multiple environments. Recently, we have encountered issues related to the maximum number of roles being created, which has necessitated increasing our role limits.
Our Situation:
We are hitting the maximum role limits in some environments.
We need to understand how to better manage the number of roles that are created, needed, managed, and potentially unused by Seedfarmer.
Our Questions:
What are the best practices for managing the number of roles created by Seedfarmer to avoid hitting the role limit?
Are there any specific strategies or configurations within Seedfarmer that can help reduce the number of roles created?
How can we identify and manage unused or redundant roles that are generated by Seedfarmer?
Are there any tools or scripts recommended for auditing and managing roles in Seedfarmer environments?
Any other suggestions or guidance to help us efficiently manage roles in our Seedfarmer deployments?
Thank you in advance for your assistance.

Hi @jiwachow
Thanks for your issue.  SeedFarmer (SF) was designed to create dedicated roles with least privilege policies.  There are no unused or redundant roles created by SF.
Since IAM is global per account, there will be:

one toolchain role in the account that acts as the toolchain
one deployment role in each account that modules are deployed in
one role per module per deployment in each account

The toolchain roles and deployment roles are nominal in count (they do not scale up).  But, there is a one-to-one role mapping per module created.  That means if you have a deployment that has 20 modules, there will be 20 roles created.  If you then use that same manifest and change the deployment name and deploy again (a unique deployment) you will have 20 more roles created.  They delete on module destroy.
You can filter on the roles in the account in IAM as all seedfarmer roles tied to modules have a distinct pattern:

seedfarmer--toolchain-role
seedfarmer--deployment-role
---* for all the roles that modules leverage for deployment

This is only for SF deployments, if there are roles that the modules themselves create SF is not aware of nor manages them (the actual modules create them and manage them).
We discourage the practice of having a module reuse a role for deployment.  If a module leverages the modulestack.yaml that gets attached at runtime, that policy should be limited in privileges to only what that module needs to deploy.  If we begin to combine the permissions (say create an uber-role) that defeats the security profile design.
If you are hitting role limits in a single account, perhaps consider another AWS account to use in parallel (SF supports multi-account deployments).  Also, you may want to look at your design and determine if a single module is doing the proper level of infrastructure management (ie. could functionality spread across 3 modules be condensed to 1).
If you are working with AWS ProServe in an engagement, please reach out to the AWS lead at the engagement, and they will be able to contact us directly so we can further provide recommendations.


Tests github.com/kubernetes-incubator/cri-o/server pkg

Also maybe split those huge functions in pieces to be easily testable. Any taker? :angel:
ok  	github.com/kubernetes-incubator/cri-o/server	0.116s	coverage: 0.8% of statements

Me and @willstudy would like to help make server pkg more easily testable and add ut :)

@weiwei04 @runcom Did this ever happen?

This should be solved via #1992

Fixed in #1992


add documentation for file read/write

Hi, is there a way to use this library to read from file and write to file to boost performance? Can you help with an example - difficult to understand interface?
I was trying to use one of following libraries... it seems using IBufferReader

https://github.com/neuecc/MessagePack-CSharp
https://github.com/protobuf-net/protobuf-net

For serialize, interfaces as follows,
void Serialize<T>(IBufferWriter<byte> writer, T value);
void Serialize<T>(Stream stream, T value);

For deserialize, interfaces as follows,
T Deserialize<T>(ReadOnlyMemory<byte> buffer);
T Deserialize<T>(Stream stream);
T Deserialize<T>(ReadOnlySequence<byte> byteSequence);

Context: we were trying to serialize/deserialize large files/.net objects for which IBuffer* handling is best.

This library isn't targeting making reading/writing to files on disk faster. But it does offer adapters between Stream and PipeReader/PipeWriter. Given a FileStream (which you carefully open with useAsync: true as a parameter!) then you could use UsePipe to to acquire a PipeReader/PipeWriter to write to that file. I would suggest though that you peruse Microsoft's own documentation for these types as to how to best use them, since this library didn't invent them--it merely adapts one to the other.


ENH: Add white matter fiber tracking to sEEG tutorial

This adds white matter tracking to the sEEG tutorial so that electrode contacts in white matter can be better interpreted.

Hey @skoudoro and @pjsjongsung, no rush, but when you have a spare few minutes, would you mind reviewing this? Specifically, there's a bit of an issue in the tutorial (tutorials/clinical/20_seeg.py) where the deterministic tracking yields very short, disconnected fibers. I'm wondering if you have any advice on parameters or methods to use to get more helpful fiber tracking. Right now I have it set to find fibers within 50 mm of an sEEG contact in white matter and there's a bunch of disconnected fibers but, ideally, it would be more specific, like 3 mm, and would show the pyramidal tract in this case (LPM 1 is a contact in white matter near left premotor cortex).

Hi @alexrockhill, I will look into it later this week. Can you share your peaks (the pam5 contains the peaks). I just want to make sure that they make sense. Not sure if your issue is due to the tractography or the peaks. Last time I encounter such an issue, it was due to the peaks so the reconstruction.

Here's the pam file: https://ohsuitg-my.sharepoint.com/:u:/r/personal/rockhill_ohsu_edu/Documents/pam_example/seeg_sample_pam.pam5?csf=1&web=1&e=u9ZkMk
Thanks for offering to review, really appreciate it.

@larsoner did you do the new release? I'm still getting MNE-misc-data version 0.27 with no diffusion images

@alexrockhill it says user not in directory and I cannot access the link. Do you have another link you could share or change the permissions of that link?

I added your gmail and UI email. OHSU doesn't allow you to open up permissions unfortunately... I could have shared on Google Drive though... probably should have

https://drive.google.com/file/d/1Fr_VLIoCw-pngGtAAoYB2Wls0obG233-/view?usp=sharing

@larsoner did you do the new release? I'm still getting MNE-misc-data version 0.27 with no diffusion images

Just pinging on this when you have a second @larsoner, thanks

I cut 0.28 a few weeks ago you need to update mne/datasets/config.py to tell it to use 0.28:
https://github.com/mne-tools/mne-python/blob/5a5b4f10467664b1444873ab879f4ae5724eedbb/mne/datasets/config.py#L93
and the hash
https://github.com/mne-tools/mne-python/blob/5a5b4f10467664b1444873ab879f4ae5724eedbb/mne/datasets/config.py#L134
But the version was wrong there (said 0.27 in the dataset itself) so I just cut a 0.29 so use that instead anyway
https://github.com/mne-tools/mne-misc-data/releases/tag/0.29

My bad, I forgot I had to do that, thanks!

Here's the artifact with the rendering of the not-very-good fiber tracking https://output.circle-artifacts.com/output/job/63884b31-afc5-40be-9e27-d1d2264fba57/artifacts/0/html/auto_tutorials/clinical/20_seeg.html#sphx-glr-auto-tutorials-clinical-20-seeg-py

Huh how do you ignore the vulture stuff? It is used... cc @larsoner

https://github.com/mne-tools/mne-python/blob/main/tools/vulture_allowlist.py

Huh how do you ignore the vulture stuff? It is used...

to find this yourself next time: git grep vulture yields only 10 hits, the last of which is:
pyproject.toml:392:paths = ['mne', 'tools/vulture_allowlist.py']


Skip dask test on windows + Python 2.7.

Get nondeterministic test failures relating to dask's get_async.
I want to return to this perhaps with @mrocklin's help, but for now, I'm marking it with a skipif() on Windows and Python 2.7.

Looks like this is hitting the same bug as reported in #1095.  Perhaps it showed up because I'm running the testsuite on a new Windows VM (win 7 64 bit).

Which test fails?

On any given run, I've seen expr1, expr8, and expr9 each fail:

Expr1: sx.sum(axis=0)
Expr8: sx.sum()
Expr9: sx - sx.sum()

They all have sx.sum() in common.  This only reveals itself in python 27.

Ah, well, even though this is non-deterministic it might not be incorrect.  Perhaps the tests should be changed to use something like np.allclose?
You're right that it would be nice to to be able to guarantee order though, at least in the single-threaded case.

Perhaps the tests should be changed to use something like np.allclose?

Agreed with you here -- I'll make that change.
I misspoke when I said "failiure", I meant "error", as in, we see nondeterministic errors, sometimes `KeyErrors

Perhaps the tests should be changed to use something like np.allclose?

Agreed -- I'll make that change.
I misspoke when I said "failiure"; I should have said "error".  We see nondeterministic errors, sometimes KeyErrors, and other times (minimally informative) Blaze NotImplementedErrors ("don't know how to compute ..."), and other times all tests pass fine.  Since tests do eventually pass, I suspect the problem isn't in the eq() implementation, but something more subtle.
I haven't had time to bound the issue; given that it's not reproducible, I suspect there's some potentially deep threading issue or somesuch.  My starting point would be to see if I can demonstrate it in dask without blaze; failing that, then I'll see if it's something to do with the dask backend for blaze.
FYI, I won't be able to put much time on this for a couple of weeks, due to other priorities.


fix (MediaStore policy fix to deny anonymous user-agents access aws-cloudfront-mediastore)

Issue #, if available:
#252
Description of changes:
Modified the AWS MediaStore policy condition to only allow the AWS CloudFront user agent. Look at the bug description for more details.
https://github.com/awslabs/aws-solutions-constructs/issues/252
Tested changes to make sure they have the intended effect of only allowing access to GET request files from the Amazon CloudFront endpoint. GET requests directly to MediaStore endpoint now just return a generic 403 access denied response.
Condition: {   StringEquals: {     'aws:UserAgent': this.cloudFrontOriginAccessIdentity.originAccessIdentityName   },   Bool: {     'aws:SecureTransport': 'true' }
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: 7906c782c018d6f2c56b5c6e3079b15673b200b3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: 59dabb56048c408ae5d6aa1d1f39bb371df0a5ac
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

AWS CodeBuild CI Report

CodeBuild project: codebuildgithubautobuildPro-fkVQbXRiQi6A
Commit ID: 3c9b6f725b0635eb2df28b25f4b6d669f36cd962
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository


Possible interference with a website's buttons

The top navigation buttons don't work with the extension turned on and work with the extension turned off on this website: https://us.pycon.org/2016/

Another example includes pull down menus on this website: https://www.interviewcake.com/article/python/coding-interview-tips

Should not have declared bootstrap in the manifest. Fixed.


bug: Scoreboard scrolling works strange, when teams appearing while scoreboard is shown

You can check it by enabling
{
  "scoreboardOverrides": {
    "showTeamsWithoutSubmissions": false
  }
}

in advanced settings.

should be fixed by #32

Scaling breaks in case of new appearing teams. Refresh, obviously, fixes it

Fixed in Dhaka sprint/hackathon


Update for the j2e-pac4j library v1.2

Replace ConfigInstance by ConfigSingleton and ConfigBuilder. Slightly change the logic in DefaultAuthorizerBuilder.

For: https://github.com/pac4j/j2e-pac4j/pull/18 & https://github.com/pac4j/j2e-pac4j-demo/pull/16


Procedure fails at Step 2

Assign eDiscovery permissions
Step1 : Go to the compliance portal and sign in using an account that can assign permissions.
Step 2: In the left pane, select Permissions.
Permissions does not appear in the left pane.  (Logged in as Global Administrator.)
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: ec7c996e-5918-c91e-c10a-1e1acacb78a5
Version Independent ID: ec7c996e-5918-c91e-c10a-1e1acacb78a5
Content: Assign eDiscovery permissions in the Microsoft Purview compliance portal - Microsoft Purview (compliance)
Content Source: microsoft-365/compliance/ediscovery-assign-permissions.md
Service: o365-seccomp
GitHub Login: @robmazz
Microsoft Alias: robmazz

@MajorCatastrophe Thank you for your feedback. We have made changes to the document via PR#https://github.com/MicrosoftDocs/microsoft-365-docs/pull/11919
Once the author approves, the changes will be live.
Thanks
Sri

@MajorCatastrophe Thanks for your feedback! New issues and pull requests for this content are closed while we update this content to support upcoming feature improvements.


Release 0.0.20

@valeryan while we discuss/test #141 I'm opening this to check if you're happy if we release 0.0.20, which includes the PRs #139 and #140?

If you feel confident and comfortable doing it go ahead. The caveats is that the secret key for releasing this is tied to my publisher account and it sometimes expires. I will have to see if I can adjunct you into my publisher account so that you don't get locked out of doing a release of I am not around.

Ah, in that case maybe it's better if you release, when you have time, of course.

The secret key should be valid right now. I think I renewed it a few months ago.

Not to bug anyone, but I am eagerly awaiting this release :D

@Dekadinious no, the reminder is good. I've been dealing with a lot of work and personal stress since these comments, but I will try and get things going this week. There's also another PR (#145 ) that I need to test, that we should probably include in this release.

Is there anything I can do to help in this matter? :)

Yeah, I have been super busy with work. We are releasing new Core D&D books so I have had no time for side stuff. Sorry.

I had some time this morning, so I have created the next release: https://github.com/valeryan/vscode-phpsab/releases/tag/v.0.0.20
Not sure how long it takes to push everything to the Marketplace. I'll keep an eye on this over the course of the day.

@valeryan I drafted a new release for v.<IP_ADDRESS> and it seems the workflow failed
https://github.com/valeryan/vscode-phpsab/actions/runs/11839743232
Did I do something wrong?

I see you are already working on this here https://github.com/valeryan/vscode-phpsab/pull/151

Thanks @valeryan
@aslamdoctor can you please test if the new version fixes the spaces issue.
Thanks

@jonathanbossenger yep, its working correctly on all my "Local" projects which has space in the path. 🎉


Use quickinfo instead of a bundled language service

Fixes #14
Fixes #16
Fixes #9
Fixes #10
This completely reworks how this extension operates following a suggestion from @acutmore. Instead of running a separate language service, I send commands to VS Code's built-in TypeScript language service. This should have lower overhead and result in fewer project configuration mismatch issues.
The new high-level workflow is:

Parse the document (currently using babel, but could use tsc) to find the Identifier nodes in the visible range of your editor.
Get quickinfo for those symbols.
Highlight the ones that end with ": any" (ish)

The extension keeps track of the line ranges that it's checked in each file using a disjoint set structure to avoid repeating work as you scroll around. Changing a file invalidates this cache and forces new quickinfo requests.
I switched to Babel for parsing because I (mistakenly) thought that importing TypeScript in my extension was causing strange JSX issues (#16). The real culprit seems to have been something about the "contributes" clause in my package.json file. I've left this change in since @babel/parser is so much smaller than typescript (~2MB vs. ~20MB).
Remaining work (maybe not for this PR):

Highlight anys when the editor/extension loads. Unfortunately, this requires waiting until the language service has processed all your imports before it gives accurate results. (Until then, symbols come back with any types!) It's unclear to me whether there's some sort of "idle" or "ready" event I can listen for to find out when this happens.
Limit batch size for quickinfo requests. At the moment I request quickinfo for all visible identifiers. If you have a tall editor, there could be a lot of them! Requests are processed in a FIFO manner, so it would be better to issue 10 or 20 at a time to let the user get some requests in the queue. This would also facilitate canceling stale requests when the user edits the document.
Cache the AST. As you scroll, I'll keep re-parsing the entire document to find the identifiers in the new range. This is wasteful. Caching the last AST could be a win for large files.

Awesome!


feat: support diff functions for some properties

What's the purpose of this PR
The diff function doesn't work for properties currently.  We'd like to compare the properties in different cluster or branch.
当前 apollo 页面上的 diff 功能不支持 properties 格式的配置，不便于对比配置。 我们希望支持 properties 格式的 diff 高亮功能
Which issue(s) this PR fixes:
Fixes #
Brief changelog

compare gray version with master version when doing a full release:

compare with released version when doing a release:

support diff of properties when comparing between clusters:

Follow this checklist to help us incorporate your contribution quickly and easily:

[X] Read the Contributing Guide before making this pull request.
[X] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[ ] Write necessary unit tests to verify the code.
[X] Run mvn clean test to make sure this pull request doesn't break anything.
[ ] Update the CHANGES log.

Summary by CodeRabbit
Release Notes

New Features

Enhanced table structure for displaying differences in the configuration settings.
Introduced a new component for visual text comparison of configuration changes.

Improvements

Updated Chinese localization for clarity in publishing status messages.
Enhanced styling for better layout control in tables and text elements.
Refined logic for displaying elements in the release modal based on release type.

Bug Fixes

Refined logic for displaying release process tables, improving clarity and functionality.

Thanks for adding this great feature. Please find some comments below:

Could the diff format for non-JSON values be made clearer?

Should we add the diff feature for grayscale release?

Thanks for adding this great feature. Please find some comments below:

Could the diff format for non-JSON values be made clearer?

Should we add the diff feature for grayscale release?

A display: block style has been added to the diff span, now the diff for strings will be seperated into different lines to make it clearer
Yes the feature is now added to the grayscale release, too. Please take another look

@nobodyiam
Hi, I'm planning to put the diff into text modal instead of in the tables, as the diff looks too crowed in the table. I'll close this pull request and make a new one when it's ready.


Race condition can prevent message from being read until another message is received

I'm not sure if anyone here is very familiar with the java client, but I was doing some debugging of an intermittent issue and I think tracked it down, but I'm not sure the best way to fix it. What I think is happening is

thread1 - https://github.com/nats-io/java-nats/blob/master/src/main/java/io/nats/client/impl/MessageQueue.java#L136 is called, and no messages are available in the queue.
thread2 - https://github.com/nats-io/java-nats/blob/master/src/main/java/io/nats/client/impl/MessageQueue.java#L101 is called. A message is received, but no waiters are currently in the queue, so nothing is signaled.
thread1 - Our waiter is eventually added, and https://github.com/nats-io/java-nats/blob/master/src/main/java/io/nats/client/impl/MessageQueue.java#L154 is called as our waiter wasn't in the list when the message was received, so we weren't able to be notified and our thread is parked, and our subscriber never receives a message. The message is pulled off the queue when another message arrives to notify the waiter, but sometimes that doesn't occur for several seconds, leaving our client receiving a timeout for the first message

I can reproduce the issue with this patch and the testPopBeforeTimeout test. The best case scenario would be the message is received as soon as the sleep in waitForTimeout is over, instead the test spins for a long, long time. (Sorry for the line endings)
diff --git a/src/main/java/io/nats/client/impl/MessageQueue.java b/src/main/java/io/nats/client/impl/MessageQueue.java
index 95e8fce..100bb6a 100644
--- a/src/main/java/io/nats/client/impl/MessageQueue.java
+++ b/src/main/java/io/nats/client/impl/MessageQueue.java
@@ -149,6 +149,8 @@ class MessageQueue {
                     }
                 }

+                Thread.sleep(5000);^M
+^M
                 waiters.add(t);
                 if (timeoutNanos == 0) {
                     LockSupport.park();
@@ -294,4 +296,4 @@ class MessageQueue {

         this.queue.addAll(newQueue);
     }
-}
\ No newline at end of file
+}^M
diff --git a/src/test/java/io/nats/client/impl/MessageQueueTests.java b/src/test/java/io/nats/client/impl/MessageQueueTests.java
index eadd84a..07aeeb4 100644
--- a/src/test/java/io/nats/client/impl/MessageQueueTests.java
+++ b/src/test/java/io/nats/client/impl/MessageQueueTests.java
@@ -109,7 +109,7 @@ public class MessageQueueTests {
         t.start();

         // Thread timing, so could be flaky
-        NatsMessage msg = q.pop(Duration.ofMillis(200));
+        NatsMessage msg = q.pop(Duration.ofMillis(2000000));^M
         assertNotNull(msg);
     }

@@ -519,4 +519,4 @@ public class MessageQueueTests {
         q.filter((msg) -> {return true;});
         assertFalse(true);
     }
-}
\ No newline at end of file
+}^M

Just to confirm - this was a rare - super annoying - hard to find issue? I am trying to think of a way to reproduce without editing the message queue and it looks super sneaky and annoying. Hats off for finding it.
What is the NATS pub/sub/msg level of how this is happening?

It happens most often in our tests, as they can be fairly single threaded and request/response oriented, so this happening once fails them. There are a fairly large number of tests so the odds of at least one failing is relatively high.
The client sends a message to nats, nats sends a message to a subscriber. Then quite awhile later the subscriber sends a message back to nats, and nats sends that message to the client. Something like


Another anti-detection series

We have lots of anti-detection issues at the moment, and plan that trend will continue for quite a while. We all must realize progress can sometimes be slow. Every addition creates new lines of thought, those lines of thought help us to create a more stable and useful system. That being said, I'd like to go over my thoughts thus far with anti-detection. I won't go too in-depth, as this is a public forum and we won't stay ahead if their devs can simply read our thoughts. Series of thoughts, in no particular order:

Altitude isn't all the important, as long as we can mimic the responses whatever device we are sending and it does not measure altitude.
If we jump great distances, bans seem to be a function of distance and time. I've waited X hours of travel time to try a new location and not been softbanned (Takes me 45m to drive there, an hour later I can teleport there).
If we stand out too much over periods of time (catches/hr, etc).
We don't use realistic times between different actions. Minus network delays, we should map out the min sleep time for all actions and stick to it. Leveling slower and not getting banned is obviously a better result.
They have the results from many real players. When was the last time you really played and ignored 50 stops before returning? We need our bot to play how people play.  How often you exited to respond to that email, phone call etc.
Some methods of detection involve too much work on their systems, so all detection methods should be rather simplistic. From what I can tell, the quota bans do not happen immediately, but I expect they're pretty automated - but do not kick in right away as it may be obvious, or they have a separate system run every so often. The ban "waves" seem to be flagged and reviewed accounts.
We should continue to make all task's wait times configurable, but now the avg app wait time (recommending it as a minimum).
As they have a wealth of legit player information, it is not unlikely they could run algorithms against they retained data. For example, players who went from stop A to stop B had a gps variance of X and fetch all players outside the bounds of variance X.
Simplest first - While we can come up with grand schemes on how bots could be detected, they will probably use the easiest first (except not sending API requests back!). Their X stops and pokemon caught per day is a perfect example. It's a very easy thing to implement.
Too many things over time (distance walked, consecutive time played, stops visited, pokemon encountered, etc).
Make sure we're simulating the same data the app sends. We can expect they will start verifying more packets or lack there of. They probably won't scan for this in real time, but run db scans every so often to look for bot trends. It'd clear they have criteria to find us. I think this explains the mass bans (somewhat mentioned above); They have thresholds to automatically flag folks then features to search flagged and suspicious accounts (that do not yet meet softbans) and flag them for ban. Pick a date then ban all ban flagged accounts. This method helps hide the banning criteria.  to reduce server load, then a much lower trigger for an actual ban.

I think it's difficult to out smart them. They have ways to find out.

If we do the job well enough, it's all just packets traveling between a client and a server. After the system, gets strong enough, detection takes too much processing power on their end. The next step is to be like MapleStory and update the API every month or so, causing programmers to have to hunt down all the values again.

"simulating the same data the app sends"
So whats are the timing and order of apis call on the app?
Exaple If app sent gps position ever 53s , or ask inventary before spin and after catch ... Bot must do rhe same. But who know

Take a look when we play with same IP for more than 1 characters
for example :
When u play using 1 connection (Thetering) from a phone....that mean your IP connection should be same between 1 character to another, same time, same connection and sure should be same path.
same path mean same fort list to spin, also same pokemon to catch...no matter which one is first.
Now, the bot able to randomize your walk speed, has capability (task) "FollowPath", etc.
But, the result is, even when you start 2 characters on same location, and set them to same path, even when you set lure_attraction : false. On view minutes / hours, those characters will bring up different result (forts or pokemon).
Now the question is, will you play in one IP but take different path (e.g one to north, and south for other)....i believe this behavior of bot might lead you to be suspected as bot. (especially for you who play with more than 1 character like me, lol).
so far what i did :

False lure_attraction
FollowPath mode : loop
minimizing path scope (about 500 m - 1km) divide to 4 altitude.
Immediately sign out the bot if my characters range to far (of course i know the place as well as play on foot).
keep delete last-location and recents-forts before start the bot.
wait for 10 - 20 minutes before restart the bot.

If just, the bot able to :

take exactly same action between 2 accounts which has same spawn location, same path, and absolutely same IP.  I remember first time i use this bot, exactly something good imo. same location to start, and walk with same path, spun same forts, and catch same pokemon, no matter vanish or catch.

(if point no 1 not possible ) pause the bot on our command (not sign out Ctrl+C in terminal), so  we can wait the other account arrive on that pokestop then walk together.

Or maybe was my mistake on set the config so the bot walk randomly (not exactly same between 2 characters)...

What's wrong with multiplayere on same ip ? No public/free IP ? Campus ?

@mjmadsen I get where you are going with this thread, but why choose issues on the github? We are trying to keep this clean for actual bugs/issues with the bot, and feature requests. I can see the linkage if your list has actual outcomes for each point (which should be in their own respective FR), but a thread like this is going to go nowhere and is going to eventually be closed. Can I suggest this conversation is moved to the slack and if you feel there isn't a channel already created for this kinds of discussion, create one

With enough samples of 'legit' and 'bot' log data, Niantic can code a simple neural network classifier.. continuously improve their model with more manually tagged 'legit'/'bot' logs.. and boom.. super accurate real-time bot detection. The NN would detect patterns we'd be unlikely to see..

They can barely keep stable game servers, let alone a neural network :p

@k4n30 Sorry this is not the best medium. Closing.


Cdr changes

I made some updates based on feedback from CDR. I also fixed some bugs and cleaned up some code that I saw while I was working on this. The main changes I made are:

removed dashboard notify task
changed some shutoff limits
removed some commented out code
moved the voltage, temperature, and current CAN message creation to after determining if the battery is safe

I just wanna make sure that our plan was to decrease both temperature thresholds and not just one. I was under the impression that we were trying to have it at 2.55 for charging and 2.9 for discharging so we would have a larger gap between the two. If we want them both to have the same gap but just at a lower threshold then I think that is fine too.

I just wanna make sure that our plan was to decrease both temperature thresholds and not just one. I was under the impression that we were trying to have it at 2.55 for charging and 2.9 for discharging so we would have a larger gap between the two. If we want them both to have the same gap but just at a lower threshold then I think that is fine too.

The plan was to decrease both of the voltage thresholds, so we can use more of our battery's charge. The gap is slightly smaller, but I think it should still be fine, since the open circuit voltage will be greater than the voltage when we are powering the motor, so we shouldn't see a drop of 0.15V after tripping the BPS


Fixed #31485 -- Updated admin's jQuery to 3.5.1.

ticket-31458

buildbot, test on selenium.

We should wait 2-3 weeks before merging this patch because jQuery 3.5.1 may appear in the meantime.

buildbot, test on selenium.

Updated to jQuery 3.5.1.

buildbot, test on selenium.

buildbot, test on selenium.


login() error callback not being called

bot.login('[USERNAME]', '[PASSWORD]', function(error, token) {
    if (error) {
        console.log('Problem occurred while logging in! ' + error);
        return;
    }
    console.log('-----------------------------------------------------------------------------');
    console.log('Token: ' + token);

});

During the Discord outage a couple minutes ago. Starting up my bot had it just sit there instead of the error callback being called telling me it had a problem logging in.

dupe of #105


Mails are not send due to "authentification failed"

Describe the bug
After some time I'm getting errors:

email.0 | 2022-08-10 10:01:02.369 | error | (COMPACT) Error 535 5.7.8 Error: authentication failed:

When entering settings and pressing button "Test mail" a mail is correctly send (without changing any options). So, the error cannot result from spam protection of mail provider I'd say (all-inkl.com).
When restarting adapter it works some time until the error comes up again.
To Reproduce
Just wait some time
Expected behavior
Mails should always be sent
Versions:

Adapter version: 1.1.3
Node version: v16.16.0
Operating system: Raspi OS (bullseye)

Additional context
Was also occuring under Stretch and without compact mode.

same here. do you found a solution?

Unfortunately no. I installed WhatsApp adapter to be able to inform, if mail should break with that error. It works and since then, I got no further errors. Shure it'S not the solution, but it seems that this error comes and goes.
Perhaps some issue when having too less memory or problems with special characters or a codepage issue...

Hm ... any new insights? because why it should work some time and then not?

No new issues since August. Seems that error has disappeared

so close?

Yes, if it will occur again, we can reopen it


Allow specifying that query return type can't be nil

On this query I would like to specify that the user that it returns can't be nil.
graphql do
  type :user

  queries do
    read_one :current_user, :current_user
  end
end

Do you have any thoughts about how you would want that implemented and I can open a PR for it?

This has been completed.


fix(gateway) handle NodePort addresses

What this PR does / why we need it:
Ensure that the controller's desired addresses for Services with no external addresses (such as NodePorts) match the Gateway representation of no addresses. This ensures that the address update check does not attempt to proceed with an unnecessary update every reconcile, which would prevent the Gateway from becoming ready.
Add an E2E test that confirms Gateways work with NodePorts.
Which issue this PR fixes:
Fix #2808
Fix #2823
Special notes for your reviewer:
E2E run: https://github.com/Kong/kubernetes-ingress-controller/actions/runs/2871403290
PR Readiness Checklist:
Complete these before marking the PR as ready to review:

[x] ~the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR~ this was a bug introduced by unreleased code from https://github.com/Kong/kubernetes-ingress-controller/pull/2797

This is currently affected by the failure that https://github.com/Kong/kubernetes-ingress-controller/pull/2824 will fix.
2022-08-16T20:24:15.5342094Z     helpers_test.go:290:
2022-08-16T20:24:15.5343441Z         	Error Trace:	/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/helpers_test.go:290
2022-08-16T20:24:15.5345023Z         	            				/home/runner/work/kubernetes-ingress-controller/kubernetes-ingress-controller/test/e2e/features_test.go:337
2022-08-16T20:24:15.5345626Z         	Error:      	Condition never satisfied
2022-08-16T20:24:15.5346129Z         	Test:       	TestDeployAllInOneDBLESSGateway
2022-08-16T20:24:15.5347559Z     features_test.go:286: cleaning up environment for cluster 14e300ea-8260-41ba-9b37-7493e874583f
2022-08-16T20:24:17.0472065Z --- FAIL: TestDeployAllInOneDBLESSGateway (396.49s)

On second thought, looking at that failure, configuring the route to strip paths makes more sense than intentionally causing a 404 and scraping the server header, so I've added that change here.

On second thought, looking at that failure, configuring the route to strip paths makes more sense than intentionally causing a 404 and scraping the server header, so I've added that change here.

OK, I did not find that we have an annotation to do the strip path of HTTPRoute. I think we do not need to put #2824 as a blocker of this.


DataDome bot and online fraud protection plugin

Description
What did you change and why?
Add DataDome DataDome bot and online fraud protection plugin to hub documentation.
Testing instructions
Netlify link:
Checklist

[ ] Review label added
[ X ] PR pointed to correct branch (main for immediate publishing, or a release branch: e.g. release/gateway-3.2, release/deck-1.17)

@laurodd can you please move /app/_hub/datadome/kong-plugin-datadome/_index.md into /app/_hub/datadome/kong-plugin-datadome/overview/_index.md, and add the following to the very top of the file:
---
nav_title: Overview
---

We've had to do some refactoring of the plugins since you submitted this PR, so it isn't building in the preview.

@laurodd @DaniellaFreese sorry for the mess, we're still in trial-and-error mode here. The schema was formatted incorrectly, and the image was in the wrong directory (the latter is another recent change in the platform).
The next deploy should now work (it was working for me locally).
Still need a couple of things for the doc itself:

I formatted the schema but it needs descriptions for each property. Right now it's still generating empty description fields:

The protocols field in the schema is currently listing http and https as the supported protocols. I just put those in as defaults. Please add in all of the protocols that your plugin supports.

Latest deploy worked:
https://deploy-preview-5995--kongdocs.netlify.app/hub/datadome/kong-plugin-datadome/

hello all, should I change something (smoke tests are failing) or this is maybe linked with the new plugin doc structure?
Pls let me know if I can help :)


[DOC] Typo correction in getting started guide

Reference Issues/PRs

Does your contribution introduce a new dependency? If yes, which one?

Any other comments?

Hey ! Thanks for contributing, I think there is a confusion on the n_timepoint thing. We never used m_timepoint, the second occurence might be a relicate of the time where we also use n_timestamps. The double occurrence of n_timepoint here is must be due to a mass replace of n_timestamps by n_timepoints and can be deleted

Hi @baraline,
thanks for the feedback, I will remove the second occurrence!

It is good to fill out the initial PR as best you can. You can delete template bits if they do not apply. Good practice to link to issue if you can, i.e. "Fixes #1841" which will close that issue on merge.


Welcome page

Is your feature request related to a problem? Please describe.
I would like to open my app and view a summary of my financial health
Describe the solution you'd like
I would like to open my app e view my name, my balance, last expenses and a month summary
Additional context
Should be like spec

Working on spec


Fix check for self-signed certs in EncodeX509Chain

See also https://github.com/jetstack/cert-manager/issues/4142
EncodeX509Chain checked for self-signed certs by comparing the subject
and issuer of the cert in question, which is invalid since it's
perfectly fine for those to match.
the correct behavior is to use cert.CheckSignatureFrom(cert). this bug
was exposed in 1.4 when ParseSingleCertificateChain started using
EncodeX509Chain in the critical path of several issuers; when end-users
had leaf certificates with subjects matching their issuer's subject, the
bug was triggered.
includes tests for EncodeX509Chain, a test for
ParseSingleCertificateChain, and some defence-in-depth measures
Fix check for self-signed certificates in EncodeX509Chain which broke certs whose subject DN matched their issuer's subject DN

/kind bug

/lgtm
/cherry-pick release-1.4

/test pull-cert-manager-bazel
almost certainly a flake