unable to reproduce the paper

hi,
I tried to reproduce the paper.i have downloaded the dataset from mcyt.there is a format issue with the dataset image format. the format they have given is .fpg .what should i do. please do help.currenty doing this paper as my PG final year project.please do help

Can you please confirm where did you obtain the MCYT dataset? The link from http://atvs.ii.uam.es/atvs/mcyt75so.html should give you the dataset in the right format

Yes its mcyt dataset.
Thanks & Regards
Achu Lekshman A

From: Luiz Gustavo Hafemann<EMAIL_ADDRESS>Sent: Monday, February 3, 2020 9:31:16 PM
To: luizgh/adversarial_signatures<EMAIL_ADDRESS>Cc: ack666<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [luizgh/adversarial_signatures] unable to reproduce the paper (#1)
Can you please confirm where did you obtain the MCYT dataset? The link from http://atvs.ii.uam.es/atvs/mcyt75so.html should give you the dataset in the right format
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/luizgh/adversarial_signatures/issues/1?email_source=notifications&email_token=AJHPX2VA2SQ6LL2OW2UGRJDRBA5UZA5CNFSM4KNV4CZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKUME4Q#issuecomment-581485170, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJHPX2SKBEL72FHNHMRUGQ3RBA5UZANCNFSM4KNV4CZA.

Did you download it from the link I sent above? Can you please post some stats of the dataset you got (number of users, number of genuine and forgeries per user, file format)?

I had requested them for the MCYT-75 offline signature database. They had sent me MCYT 100 Offline signatures instead of the MCYT -75 dataset. The dataset they had sent is of .fpg format, encrypted i think, along with an FPG_Signature_View application.
The dataset contains 25 forged and 25 originals for each user and a total of 100 users.
Thanks & Regards,
Achu Lekshman A

From: Luiz Gustavo Hafemann<EMAIL_ADDRESS>Sent: 03 February 2020 16:07
To: luizgh/adversarial_signatures<EMAIL_ADDRESS>Cc: ack666<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [luizgh/adversarial_signatures] unable to reproduce the paper (#1)
Did you download it from the link I sent above? Can you please post some stats of the dataset you got (number of users, number of genuine and forgeries per user, file format)?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/luizgh/adversarial_signatures/issues/1?email_source=notifications&email_token=AJHPX2WICGR67IUBYGRZ43TRBA6KPA5CNFSM4KNV4CZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKUM2YQ#issuecomment-581487970, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJHPX2QYMOSBI6AYOHQRASDRBA6KPANCNFSM4KNV4CZA.

also getting erroers while installing the sigver package
Exception:
Traceback (most recent call last):
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/commands/install.py", line 353, in run
wb.build(autobuilding=True)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
self.requirement_set.prepare_files(self.finder)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/req/req_set.py", line 554, in _prepare_file
require_hashes
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/req/req_install.py", line 278, in populate_link
self.link = finder.find_requirement(self, upgrade)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/index.py", line 465, in find_requirement
all_candidates = self.find_all_candidates(req.name)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/index.py", line 423, in find_all_candidates
for page in self._get_pages(url_locations, project_name):
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/index.py", line 568, in _get_pages
page = self._get_page(location)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/index.py", line 683, in _get_page
return HTMLPage.get_page(link, session=self.session)
File "/home/ack666/signature_test/signature/lib/python3.6/site-packages/pip/index.py", line 795, in get_page
resp.raise_for_status()
File "/home/ack666/signature_test/signature/share/python-wheels/requests-2.18.4-py2.py3-none-any.whl/requests/models.py", line 935, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pypi.org/simple/visdom-logger/

Hmm, I will need to take a look on the issue with visdom logger. For now, you can clone the following repository, and install it with pip install .: https://github.com/luizgh/visdom_logger.
Regarding MCYT, unfortunately I am not familiar with the format you described. I suggest you reach back to them asking for the MCYT-75 version.

File "scripts/train_cnns.sh", line 7
python -m sigver.featurelearning.train --dataset-path  ~/Datasets/gpds_170_242.npz
^
SyntaxError: invalid syntax
getting this error.

unable to run example.py. getting error
HTTPError                                 Traceback (most recent call last)
 in
7 if not Path('data/dataset_joao.npz').exists():
8     request.urlretrieve('https://storage.googleapis.com/luizgh-datasets/datasets/dataset_joao.npz',
----> 9                         'data/dataset_joao.npz')
10     request.urlretrieve('https://storage.googleapis.com/luizgh-datasets/datasets/mcyt_train_signet_features.npz',
11                         'data/mcyt_train_signet_features.npz')
/usr/lib/python3.6/urllib/request.py in urlretrieve(url, filename, reporthook, data)
246     url_type, path = splittype(url)
247
--> 248     with contextlib.closing(urlopen(url, data)) as fp:
249         headers = fp.info()
250
/usr/lib/python3.6/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
221     else:
222         opener = _opener
--> 223     return opener.open(url, data, timeout)
224
225 def install_opener(opener):
/usr/lib/python3.6/urllib/request.py in open(self, fullurl, data, timeout)
530         for processor in self.process_response.get(protocol, []):
531             meth = getattr(processor, meth_name)
--> 532             response = meth(req, response)
533
534         return response
/usr/lib/python3.6/urllib/request.py in http_response(self, request, response)
640         if not (200 <= code < 300):
641             response = self.parent.error(
--> 642                 'http', request, response, code, msg, hdrs)
643
644         return response
/usr/lib/python3.6/urllib/request.py in error(self, proto, *args)
568         if http_err:
569             args = (dict, 'default', 'http_error_default') + orig_args
--> 570             return self._call_chain(*args)
571
572 # XXX probably also want an abstract factory that knows when it makes
/usr/lib/python3.6/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
502         for handler in handlers:
503             func = getattr(handler, meth_name)
--> 504             result = func(*args)
505             if result is not None:
506                 return result
/usr/lib/python3.6/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
648 class HTTPDefaultErrorHandler(BaseHandler):
649     def http_error_default(self, req, fp, code, msg, hdrs):
--> 650         raise HTTPError(req.full_url, code, msg, hdrs, fp)
651
652 class HTTPRedirectHandler(BaseHandler):
HTTPError: HTTP Error 404: Not Found

please do help

@achulekshman Thanks for catching that. I moved the data to another location (on google drive). You can download it from these links:
https://drive.google.com/open?id=1MPNJVVQXZwz38dmeqIeyditIUxTpFO2d
https://drive.google.com/open?id=1r-5lnJtChaAa8R4ocE1ZyorTdTiAqlQA
https://drive.google.com/open?id=1KffsnZu8-33wXklsodofw-a-KX6tAsVN

PS: I spot checked the links on the Readme file and they look good, so it seems that the problem is only on the code for "examples" to download it

mathlab version used.??
how to integrate it with python

I used version 2013a, but more recent versions should work as well. The Python integration can be installed following this link: https://www.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html

i am having trouble with the api thing.getting error while installing
ack666@ack666:/usr/local/MATLAB/R2019b/extern/engines/python$ python3 setup.py install
running install
running build
running build_py
creating build
error: could not create 'build': Permission denied

also tried
ack666@ack666:/usr/local/MATLAB/R2019b/extern/engines/python$ python3 setup.py install --user
running install
running build
running build_py
creating build
error: could not create 'build': Permission denied
still the same issue.
couldn't find a fix online

I think I may have had this problem, but it was a long time ago that I installed it, so I don't remember it exactly. I think I made a copy of the "engines/python" folder to my HOME, and then built it. It is probably trying to create a build folder on /usr/local ..., which your user don't have access to.

that didnt work.
still struck over here
ack666@ack666:~/extern/engines/python$ python3 setup.py install
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/matlab
copying dist/matlab/mlarray.py -> build/lib/matlab
copying dist/matlab/init.py -> build/lib/matlab
copying dist/matlab/mlexceptions.py -> build/lib/matlab
creating build/lib/matlab/engine
copying dist/matlab/engine/enginehelper.py -> build/lib/matlab/engine
copying dist/matlab/engine/basefuture.py -> build/lib/matlab/engine
copying dist/matlab/engine/matlabengine.py -> build/lib/matlab/engine
copying dist/matlab/engine/futureresult.py -> build/lib/matlab/engine
copying dist/matlab/engine/fevalfuture.py -> build/lib/matlab/engine
copying dist/matlab/engine/init.py -> build/lib/matlab/engine
copying dist/matlab/engine/engineerror.py -> build/lib/matlab/engine
copying dist/matlab/engine/matlabfuture.py -> build/lib/matlab/engine
copying dist/matlab/engine/enginesession.py -> build/lib/matlab/engine
creating build/lib/matlab/_internal
copying dist/matlab/_internal/mlarray_utils.py -> build/lib/matlab/_internal
copying dist/matlab/_internal/mlarray_sequence.py -> build/lib/matlab/_internal
copying dist/matlab/_internal/init.py -> build/lib/matlab/_internal
error: The installation of MATLAB is corrupted.  Please reinstall MATLAB or contact Technical Support for assistance.

Unfortunately I don't think I can help with that. Maybe it's better to contact MATLAB support for its installation. I did notice, however, that in that link that I reference (https://www.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html), they state

You might need administrator privileges to execute these commands.

So maybe you need root to install it for you

i had rectified matlab related issue

now getting another error.
C:\Users\Achu Lekshman A\Documents\adversarial_signatures-master\adversarial_sig
natures>python extract_features.py --data-path ~/Datasets/mcyt_170_242_otsuafter
.npz --weights-path ~/runs/signet/model_last.pth --save-path-signet ~/runs/adv/
features/mcyt_signet.npy --save-path-lbp ~/runs/adv/features/mcyt_lbp.npy
usage: extract_features.py [-h] --data-path DATA_PATH --weights-path
WEIGHTS_PATH --save-path-signet SAVE_PATH_SIGNET
--save-path-lbp SAVE_PATH_LBP
extract_features.py: error: the following arguments are required: --weights-path
, --save-path-lbp

do i need to create run folder followed by signet and copy the model_last.pth.
is that the issue here.

@achulekshman that's quite strange. It seems that the error is coming from python's argparse, not recognizing the command line arguments. Can you copy the entire error message here?

Also, are you running on Windows? I only tested this on Linux, if you have access to a linux machine it would be preferable

C:\Users\Achu Lekshman A\Documents\adversarial_signatures-master\adversarial_sig
natures>
C:\Users\Achu Lekshman A\Documents\adversarial_signatures-master\adversarial_sig
natures>python extract_features.py --data-path ~/Datasets/mcyt_170_242_otsuafter
.npz --weights-path ~/runs/signet/model_last.pth --save-path-signet ~/runs/adv/
features/mcyt_signet.npy --save-path-lbp ~/runs/adv/features/mcyt_lbp.npy
usage: extract_features.py [-h] --data-path DATA_PATH --weights-path
WEIGHTS_PATH --save-path-signet SAVE_PATH_SIGNET
--save-path-lbp SAVE_PATH_LBP
extract_features.py: error: the following arguments are required: --weights-path
, --save-path-lbp
this is the error

as i am having error with mathlab in ubuntu ,migrated to windows 8.1

ur code works well in windows also.

Hmm, I couldn't replicate the issue. That is strange, looks like argparse is not processing the command line arguments properly. Also, are you sure you can use "~" to refer to HOME in windows? I'd try with the full path to the files/folders to see if it works

i switched back to linux nd noww
ack666@ack666:~/signature_test$ python3 extract_features.py --data-path ~/Datasets/mcyt_170_242_otsuafter.npz --weights-path ~/runs/signet/model_last.pth --save-path-signet ~/runs/adv/features/mcyt_signet.npy --save-path-lbp ~/runs/adv/features/mcyt_lbp.npy
terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc
Aborted (core dumped)
this error....
please do help.i'm actually struck .i didn't know the flow of ur code.im actually new to this.

ack666@ack666:~/signature_test/scripts$ ./process_datasets.sh
Processing dataset
Number of users: 75
Allocated x of shape: (2250, 170, 242)
100%|███████████████████████████████████████████| 75/75 [02:19<00:00,  1.86s/it]
Processing dataset
Number of users: 55
Allocated x of shape: (2640, 170, 242)
100%|███████████████████████████████████████████| 55/55 [02:32<00:00,  2.77s/it]
File "process_dataset.py", line 13
def process_dataset(dataset: IterableDataset,
^
SyntaxError: invalid syntax
terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc
./process_datasets.sh: line 14: 19184 Aborted                 (core dumped) python3 extract_features.py --data-path ~/Datasets/mcyt_170_242_otsuafter.npz --weights-path ~/runs/signet/model_last.pth --save-path-signet ~/runs/adv/features/mcyt_signet.npy --save-path-lbp ~/runs/adv/features/mcyt_lbp.npy
terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc
./process_datasets.sh: line 18: 19194 Aborted                 (core dumped) python3 extract_features.py --data-path ~/Datasets/cedar_170_242_otsuafter.npz --weights-path ~/runs/signet/model_last.pth --save-path-signet ~/runs/adv/features/cedar_signet.npy --save-path-lbp ~/runs/adv/features/cedar_lbp.npy
terminate called after throwing an instance of 'std::bad_alloc'
what():  std::bad_alloc
./process_datasets.sh: line 22: 19204 Aborted                 (core dumped) python3 extract_features.py --data-path ~/Datasets/gpds_170_242_otsuafter.npz --weights-path ~/runs/signet/model_last.pth --save-path-signet ~/runs/adv/features/gpds_signet.npy --save-path-lbp ~/runs/adv/features/gpds_lbp.npy

Traceback (most recent call last):
File "cnn_attacks.py", line 14, in
from attacks.attack_utils import rmse
File "/home/ack666/signature_test/scripts/attacks/attack_utils.py", line 7, in
from l2_attack import CarliniL2
ModuleNotFoundError: No module named 'l2_attack'

please do help.

Hello @achulekshman
The bad_alloc error seems to indicate out of memory. The issue above with the "ModuleNotFoundError" is because you need to download this repo: https://github.com/carlini/nn_robust_attacks. This was supposed to be on the README, I will update it. You can download it to the same folder as "attacks", or anywhere and update your PYTHONPATH environment variable.
I am sorry you are having so much trouble with this code. It's definitely not for beginners, specially due to the python-matlab integration that makes things pretty complicated. This is only necessary for attacking the LBP-based systems. If you just want to attack CNN-based systems it should be OK to run, at least for the CEDAR and MCYT datasets that are still public.

I just need to visualize the attack and countermeasures.type 1 and 2 attack is there at example file.please do add code to visualize the code. I am not able to work out even a small thing out of this code.not even getting the flow of the code.please do help. currently doing my thesis work on this paper. struck @ implementation for over 4 months.I am using virtual box to run ubuntu.

@achulekshman if you want to just run these attacks for the CNN-based systems, that should be very straightforward. Please see https://github.com/luizgh/adversarial_signatures/blob/master/example.ipynb which does exactly that: trains a model, runs the two types of attacks, prints the amount of noise (RMSE) and visualizes them. I just followed through the steps now on another machine and everything worked. I also updated the links on the notebook, with the links I sent you before.
Now, if you want to run attacks on LBP, then with the current code it's much more complicated because the LBP version is implemented in matlab, and this integration makes things more cumbersome. I chose this implementation because it was what other researchers were using and I want a fair comparison. However, you may find other LBP implementations in python, that would greatly simplify things. Anyway, as I mentioned before, running the tests on the CNN-based model is much simpler, so I would start with that. I hope this helps.

yes that code in example.ipynb is working fine before.but its not showing anything about the defense like Ens Adv training or Madry defenses.why i am not able to run the rest of ur code.i had followed step by step things in ur read me file.its getting a bunch of errors.

What error are you facing now?
As I mentioned before, the matlab integration complicates a lot. If you just want experiments on the CNN-based models, I suggest that you remove the code that runs the LBP stuff.
To help you with this I created another branch: https://github.com/luizgh/adversarial_signatures/tree/no_lbp
I updated the code/scripts to only run on MCYT and CEDAR (that you should still be able to download), and run only the CNN-based experiments. I only set it up to run the Perfect Knowledge experiments, but you can use the same idea to run the Limited Knowledge experiments.
I noticed that newer versions of foolbox break some of the code, please install the following specific package versions:
foolbox==1.8.0
randomgen==1.6.2

For the carlini attack, make sure to put the "nn_robust_attacks" folder in your PYTHONPATH.
I re-ran the code on my computer (after the changes mentioned above), and now all seems to work (I didn't wait for the attacks to finish as this takes a long time, but I ran a few users with a lower number of iterations).
If you want to use the updated scripts "as-is", add data to these directories:
~/Datasets/CEDAR
~/Datasets/MCYT-ORIGINAL/MCYToffline75original
Add the model to this directory:
~/dev/adversarial_signatures/models/signet.pth
Then, on the scripts folder, run:
bash process_datasets.sh
bash choose_files.sh
bash cnn_attacks.sh

Please note that the models trained with Ens Adv Training and the Madry defense are also listed in the Readme. You can used those models in the example.ipynb directly, or as part of the "choose_files" and "cnn_attacks" scripts.
I hope this helps

Defaulting to user installation because normal site-packages is not writeable
ERROR: Could not find a version that satisfies the requirement randomgen==1.6.2 (from versions: 1.14, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.15.0, 1.15.1, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.18.0, 1.18.1)
ERROR: No matching distribution found for randomgen==1.6.2

did u mean  randomgen==1.16.2 .couldnt find the version  randomgen==1.6.2 .

ack666@ack666:~/adversarial_signatures-no_lbp/scripts$ bash process_datasets.sh
Processing dataset
Number of users: 75
Allocated x of shape: (2250, 170, 242)
100%|███████████████████████████████████████████| 75/75 [01:56<00:00,  1.55s/it]
Processing dataset
Number of users: 55
Allocated x of shape: (2640, 170, 242)
100%|███████████████████████████████████████████| 55/55 [02:25<00:00,  2.65s/it]
Loading data
(2250, 1, 170, 242)
Extracting SigNet features
Traceback (most recent call last):
File "extract_features.py", line 44, in
np.save(args.save_path_signet, cnn_features)
File "/usr/lib/python3/dist-packages/numpy/lib/npyio.py", line 490, in save
fid = open(file, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/ack666/dev/adversarial_signatures/data/mcyt_signet.npy'
Loading data
(2640, 1, 170, 242)
Extracting SigNet features
Traceback (most recent call last):
File "extract_features.py", line 44, in
np.save(args.save_path_signet, cnn_features)
File "/usr/lib/python3/dist-packages/numpy/lib/npyio.py", line 490, in save
fid = open(file, "wb")
FileNotFoundError: [Errno 2] No such file or directory: '/home/ack666/dev/adversarial_signatures/data//cedar_signet.npy'

what is this file mcyt_signet.npy .i have a file named mcyt_170_242_otsuafter.npz after the preprocessing step.

Yes, I meant 1.16.2.
About the error, the script is trying to write the file  '/home/ack666/dev/adversarial_signatures/data/mcyt_signet.npy'. Please create tu folder  '/home/ack666/dev/adversarial_signatures/data'

ack666@ack666:~/adversarial_signatures-no_lbp/scripts$ bash cnn_attacks.sh
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
ImportError: numpy.core.multiarray failed to import
ImportError: numpy.core.umath failed to import
ImportError: numpy.core.umath failed to import
2020-04-20 17:20:18.300626: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr
cnn_attacks.sh: line 4: 24137 Aborted                 (core dumped) python3 cnn_attacks.py --data-path ~/dev/adversarial_signatures/data/mcyt_classifiers.pickle --defense-model-path ~/dev/adversarial_signatures/models/signet.pth --save-path ~/dev/adversarial_signatures/data/mcyt_cnn_attacks_pk.pickle
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
ImportError: numpy.core.multiarray failed to import
ImportError: numpy.core.umath failed to import
ImportError: numpy.core.umath failed to import
2020-04-20 17:20:21.660184: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr
cnn_attacks.sh: line 7: 24146 Aborted                 (core dumped) python3 cnn_attacks.py --data-path ~/dev/adversarial_signatures/data/mcyt_classifiers.pickle --defense-model-path ~/dev/adversarial_signatures/models/signet.pth --save-path ~/dev/adversarial_signatures/data/mcyt_cnn_attacks_lk.pickle --lk
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
ImportError: numpy.core.multiarray failed to import
ImportError: numpy.core.umath failed to import
ImportError: numpy.core.umath failed to import
2020-04-20 17:20:24.058417: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr
cnn_attacks.sh: line 11: 24155 Aborted                 (core dumped) python3 cnn_attacks.py --data-path ~/dev/adversarial_signatures/data/cedar_classifiers.pickle --defense-model-path ~/dev/adversarial_signatures/models/signet.pth --save-path ~/runs/adv/cedar_cnn_attacks_pk.pickle
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb
ImportError: numpy.core.multiarray failed to import
ImportError: numpy.core.umath failed to import
ImportError: numpy.core.umath failed to import
2020-04-20 17:20:26.393235: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr
cnn_attacks.sh: line 14: 24164 Aborted                 (core dumped) python3 cnn_attacks.py --data-path ~/dev/adversarial_signatures/data/cedar_classifiers.pickle --defense-model-path ~/dev/adversarial_signatures/models/signet.pth --save-path ~/runs/adv/cedar_cnn_attacks_lk.pickle --lk

is this some sort of versioning issue?could not find a solution online.
i have already installed in accordance with the requirements.txt
ack666@ack666:~/adversarial_signatures-no_lbp$ pip3 install -r requirements.txt Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tqdm in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 1)) (4.39.0)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (1.13.3)
Requirement already satisfied: matplotlib>=2.0 in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 3)) (3.2.1)
Requirement already satisfied: torch>=0.4.1 in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 4)) (1.3.1)
Requirement already satisfied: torchvision>=0.2.1 in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 5)) (0.4.2)
Requirement already satisfied: visdom_logger from git+git://github.com/luizgh/visdom_logger#egg=visdom_logger in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 6)) (0.1)
Requirement already satisfied: scikit-learn in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 7)) (0.21.3)
Requirement already satisfied: scikit-image in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 8)) (0.16.2)
Requirement already satisfied: foolbox==1.8.0 in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 9)) (1.8.0)
Requirement already satisfied: randomgen==1.16.2 in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 10)) (1.16.2)
Requirement already satisfied: simanneal in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 11)) (0.5.0)
Requirement already satisfied: cleverhans in /home/ack666/.local/lib/python3.6/site-packages (from -r requirements.txt (line 12)) (3.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /home/ack666/.local/lib/python3.6/site-packages (from matplotlib>=2.0->-r requirements.txt (line 3)) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/ack666/.local/lib/python3.6/site-packages (from matplotlib>=2.0->-r requirements.txt (line 3)) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /home/ack666/.local/lib/python3.6/site-packages (from matplotlib>=2.0->-r requirements.txt (line 3)) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /home/ack666/.local/lib/python3.6/site-packages (from matplotlib>=2.0->-r requirements.txt (line 3)) (2.8.1)
Requirement already satisfied: six in /home/ack666/.local/lib/python3.6/site-packages (from torchvision>=0.2.1->-r requirements.txt (line 5)) (1.14.0)
Requirement already satisfied: pillow>=4.1.1 in /home/ack666/.local/lib/python3.6/site-packages (from torchvision>=0.2.1->-r requirements.txt (line 5)) (7.1.1)
Requirement already satisfied: visdom in /home/ack666/.local/lib/python3.6/site-packages (from visdom_logger->-r requirements.txt (line 6)) (<IP_ADDRESS>)
Requirement already satisfied: scipy>=0.17.0 in /home/ack666/.local/lib/python3.6/site-packages (from scikit-learn->-r requirements.txt (line 7)) (1.4.1)
Requirement already satisfied: joblib>=0.11 in /home/ack666/.local/lib/python3.6/site-packages (from scikit-learn->-r requirements.txt (line 7)) (0.14.1)
Requirement already satisfied: PyWavelets>=0.4.0 in /home/ack666/.local/lib/python3.6/site-packages (from scikit-image->-r requirements.txt (line 8)) (1.1.1)
Requirement already satisfied: networkx>=2.0 in /home/ack666/.local/lib/python3.6/site-packages (from scikit-image->-r requirements.txt (line 8)) (2.4)
Requirement already satisfied: imageio>=2.3.0 in /home/ack666/.local/lib/python3.6/site-packages (from scikit-image->-r requirements.txt (line 8)) (2.6.1)
Requirement already satisfied: requests in /home/ack666/.local/lib/python3.6/site-packages (from foolbox==1.8.0->-r requirements.txt (line 9)) (2.22.0)
Requirement already satisfied: GitPython in /home/ack666/.local/lib/python3.6/site-packages (from foolbox==1.8.0->-r requirements.txt (line 9)) (3.0.5)
Requirement already satisfied: setuptools in /home/ack666/.local/lib/python3.6/site-packages (from foolbox==1.8.0->-r requirements.txt (line 9)) (45.1.0)
Requirement already satisfied: cython>=0.26 in /home/ack666/.local/lib/python3.6/site-packages (from randomgen==1.16.2->-r requirements.txt (line 10)) (0.29.14)
Requirement already satisfied: wheel in /home/ack666/.local/lib/python3.6/site-packages (from randomgen==1.16.2->-r requirements.txt (line 10)) (0.34.2)
Requirement already satisfied: nose in /home/ack666/.local/lib/python3.6/site-packages (from cleverhans->-r requirements.txt (line 12)) (1.3.7)
Requirement already satisfied: pycodestyle in /home/ack666/.local/lib/python3.6/site-packages (from cleverhans->-r requirements.txt (line 12)) (2.5.0)
Requirement already satisfied: mnist~=0.2 in /home/ack666/.local/lib/python3.6/site-packages (from cleverhans->-r requirements.txt (line 12)) (0.2.2)
Requirement already satisfied: tensorflow-probability in /home/ack666/.local/lib/python3.6/site-packages (from cleverhans->-r requirements.txt (line 12)) (0.9.0)
Requirement already satisfied: tornado in /home/ack666/.local/lib/python3.6/site-packages (from visdom->visdom_logger->-r requirements.txt (line 6)) (6.0.3)
Requirement already satisfied: torchfile in /home/ack666/.local/lib/python3.6/site-packages (from visdom->visdom_logger->-r requirements.txt (line 6)) (0.1.0)
Requirement already satisfied: pyzmq in /home/ack666/.local/lib/python3.6/site-packages (from visdom->visdom_logger->-r requirements.txt (line 6)) (18.1.1)
Requirement already satisfied: websocket-client in /home/ack666/.local/lib/python3.6/site-packages (from visdom->visdom_logger->-r requirements.txt (line 6)) (0.57.0)
Requirement already satisfied: jsonpatch in /home/ack666/.local/lib/python3.6/site-packages (from visdom->visdom_logger->-r requirements.txt (line 6)) (1.25)
Requirement already satisfied: decorator>=4.3.0 in /home/ack666/.local/lib/python3.6/site-packages (from networkx>=2.0->scikit-image->-r requirements.txt (line 8)) (4.4.2)
Requirement already satisfied: certifi>=2017.4.17 in /home/ack666/.local/lib/python3.6/site-packages (from requests->foolbox==1.8.0->-r requirements.txt (line 9)) (2019.11.28)
Requirement already satisfied: idna<2.9,>=2.5 in /home/ack666/.local/lib/python3.6/site-packages (from requests->foolbox==1.8.0->-r requirements.txt (line 9)) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/ack666/.local/lib/python3.6/site-packages (from requests->foolbox==1.8.0->-r requirements.txt (line 9)) (1.25.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /home/ack666/.local/lib/python3.6/site-packages (from requests->foolbox==1.8.0->-r requirements.txt (line 9)) (3.0.4)
Requirement already satisfied: gitdb2>=2.0.0 in /home/ack666/.local/lib/python3.6/site-packages (from GitPython->foolbox==1.8.0->-r requirements.txt (line 9)) (2.0.6)
Requirement already satisfied: gast>=0.2 in /home/ack666/.local/lib/python3.6/site-packages (from tensorflow-probability->cleverhans->-r requirements.txt (line 12)) (0.2.2)
Requirement already satisfied: cloudpickle>=1.2.2 in /home/ack666/.local/lib/python3.6/site-packages (from tensorflow-probability->cleverhans->-r requirements.txt (line 12)) (1.3.0)
Requirement already satisfied: jsonpointer>=1.9 in /home/ack666/.local/lib/python3.6/site-packages (from jsonpatch->visdom->visdom_logger->-r requirements.txt (line 6)) (2.0)
Requirement already satisfied: smmap2>=2.0.0 in /home/ack666/.local/lib/python3.6/site-packages (from gitdb2>=2.0.0->GitPython->foolbox==1.8.0->-r requirements.txt (line 9)) (2.0.5)

Looks like a problem with the tensorflow installation. Try opening a python interpreter and running the following commands, one by one:
import tensorflow
import foolbox.attacks
If one of them fails, something went wrong in their installation. You may need to refer to https://www.tensorflow.org/ and https://foolbox.readthedocs.io/en/v1.8.0/ for the installation instructions.

Note that we don't use Tensorflow on this software, but it is a requirement for foolbox!
FYI, I am using the following versions
tensorflow: 1.14
foolbox: 1.8.0
numpy: 1.18.2
I would just install these exact versions first. Let me know if it works and I will update the requirements file

Versions I am using for the rest of the dependencies:
simanneal: 0.5.0
scikit-learn: 0.19.2
scikit-image: 0.16.2

ack666@ack666:~$ pip3 install tensorflow==1.14
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tensorflow==1.14 in ./.local/lib/python3.6/site-packages (1.14.0)
Requirement already satisfied: wrapt>=1.11.1 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (1.11.2)
Requirement already satisfied: google-pasta>=0.1.6 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (0.1.8)
Requirement already satisfied: astor>=0.6.0 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (0.8.1)
Collecting tensorboard<1.15.0,>=1.14.0
Using cached tensorboard-1.14.0-py3-none-any.whl (3.1 MB)
Requirement already satisfied: keras-preprocessing>=1.0.5 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (1.1.0)
Requirement already satisfied: six>=1.10.0 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (1.14.0)
Requirement already satisfied: grpcio>=1.8.6 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (1.26.0)
Requirement already satisfied: absl-py>=0.7.0 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (0.9.0)
Collecting numpy<2.0,>=1.14.5
Downloading numpy-1.18.3-cp36-cp36m-manylinux1_x86_64.whl (20.2 MB)
|████████████████████████████████| 20.2 MB 1.3 MB/s
Requirement already satisfied: keras-applications>=1.0.6 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (1.0.8)
Requirement already satisfied: termcolor>=1.1.0 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (1.1.0)
Requirement already satisfied: wheel>=0.26 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (0.34.2)
Requirement already satisfied: gast>=0.2.0 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (0.2.2)
Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0
Using cached tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB)
Requirement already satisfied: protobuf>=3.6.1 in ./.local/lib/python3.6/site-packages (from tensorflow==1.14) (3.11.2)
Requirement already satisfied: markdown>=2.6.8 in ./.local/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (3.1.1)
Requirement already satisfied: werkzeug>=0.11.15 in ./.local/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (0.16.1)
Requirement already satisfied: setuptools>=41.0.0 in ./.local/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14) (45.1.0)
Requirement already satisfied: h5py in ./.local/lib/python3.6/site-packages (from keras-applications>=1.0.6->tensorflow==1.14) (2.10.0)
ERROR: tensorflow-gpu 2.1.0 has requirement tensorboard<2.2.0,>=2.1.0, but you'll have tensorboard 1.14.0 which is incompatible.
ERROR: tensorflow-gpu 2.1.0 has requirement tensorflow-estimator<2.2.0,>=2.1.0rc0, but you'll have tensorflow-estimator 1.14.0 which is incompatible.
Installing collected packages: numpy, tensorboard, tensorflow-estimator
Successfully installed numpy-1.18.3 tensorboard-1.14.0 tensorflow-estimator-1.14.0

FYI
For the carlini attack, make sure to put the "nn_robust_attacks" folder in your PYTHONPATH
you will have to rename as the folder name might be "nn_robust_attacks_master"

i think it wrkd.. .currently performing aqttacks. .every thing seems to be perfect

seems like it's going on forever. could you please check the output mentioned below.is that some sort of issue.
Attacking User 10; Image type: random, model: model_cnn_rbf    0:00:52
FGM
Carlini
Boundary
run with verbose=True to see details
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/blended_noise.py:70: UserWarning: BlendedUniformNoiseAttack failed to draw a random image that is adversarial.
warnings.warn('BlendedUniformNoiseAttack failed to draw a'
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/base.py:148: UserWarning: BlendedUniformNoiseAttack did not find an adversarial, maybe the model or the criterion is not supported by this attack.
' attack.'.format(self.name()))
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/boundary_attack.py:211: UserWarning: Initialization failed. If the criterion is targeted, it might be necessary to pass an explicit starting point or targeted initialization attack.
'Initialization failed. If the criterion is targeted,'
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/base.py:148: UserWarning: BoundaryAttack did not find an adversarial, maybe the model or the criterion is not supported by this attack.
' attack.'.format(self.name()))
Anneal
Temperature        Energy    Accept   Improve     Elapsed   Remaining
Attacking User 10; Image type: random, model: model_cnn_linear 0:00:00
FGM
Carlini
Boundary
run with verbose=True to see details
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/blended_noise.py:70: UserWarning: BlendedUniformNoiseAttack failed to draw a random image that is adversarial.
warnings.warn('BlendedUniformNoiseAttack failed to draw a'
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/base.py:148: UserWarning: BlendedUniformNoiseAttack did not find an adversarial, maybe the model or the criterion is not supported by this attack.
' attack.'.format(self.name()))
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/boundary_attack.py:211: UserWarning: Initialization failed. If the criterion is targeted, it might be necessary to pass an explicit starting point or targeted initialization attack.
'Initialization failed. If the criterion is targeted,'
/home/ack666/.local/lib/python3.6/site-packages/foolbox/attacks/base.py:148: UserWarning: BoundaryAttack did not find an adversarial, maybe the model or the criterion is not supported by this attack.
' attack.'.format(self.name()))
Anneal
Temperature        Energy    Accept   Improve     Elapsed   Remaining
Attacking User 10; Image type: skilled, model: model_cnn_rbf   0:00:00
FGM
Carlini
this message is coming after each user.
the process is still going on.since last 17hrs.

These attacks take a lot of time, specially the black-box attacks. Each iteration requires a forward pass in the CNN, and they require thousands of iterations per attack. Btw, from the paper, the black box attacks (anneal and decision-based) performed much worse, so you may want to skip them if you don't need the results (comment out the code to run them).
The warning message from the Boundary attack is normal - it means that it failed.

hello sir even i was trying to reproduce paper,can you share the link of all the  four datasets where i can request for thank you in advance


[IDE] When multiple files with same name are opened in tabs, project name should be shown in the tab's title

When two files with the same name are opened, the titles in the editor's tab bar only show their name and it's difficult to distinguish which file comes from what project/folder.
Show the path in the tab titles of files with the same name, when we have such files opened in the editor.

Issue cloned to https://github.com/eclipse/dirigible/issues/1743

Fixed in new layout

This issue is closed as the project is no longer maintained


tls poc

We are verifying with IBM security to see if it would be appropriate to add the self-signed certificate files.  If we find it appropriate, we will make a separate PR including those.  For now, the main.go in the cmd directory will not be able to read the hardcode cert files if the "-s" flag is used.

@Christopher-Wolf-ibm Repo already contains self-signed certs https://github.com/IBM/fluent-forward-go/tree/main/fluent/client/clientfakes

@Christopher-Wolf-ibm I don't want to merge broken code. You can make  the cert paths arguments passed to main if you want. Or use the certs in the project.
Also, test certs aren't an issue any more than fake passwords in a repo are an issue. They're only applicable for local testing.


Transfer to new GitHub org

To support CNCF onbaording (https://github.com/cncf/toc/issues/556)

[ ] GitHub: move cdk8s from AWSLabs
[ ] GitHub: ensure 'thelinuxfoundation' and 'caniszczyk' are added as initial org owners, this helps us make sure we have continuity of GH ownership

Quick ping on this, thanks!

We are on it. Sorry this has been taking so long.

DONE (finally!)


Charging not starting when connecting the car to charger after charging period has started

Version of the custom_component

Version 1.0.3
Configuration
The integration is controlling charging via the car.
Describe the bug
If the charging period has already started when connecting the car to the charger, charging will not start.

Solution added in v1.3.0


Update all string return statements in codebase

Thanks to https://github.com/Raleigh-Foster we were able to find a massive issue with all return statements that return a string value....on MacOS this issue broke every part of the command but on Linux it worked fine. Thanks to this S/O article we found the solution. Essentially I need to go through the ENTIRE codebase and update all return statements that return a string/string(s) with Odins  strings.clone(valueToReturn) proc.
For an example of how I already implemented this see https://github.com/Solitude-Software-Solutions/OstrichDB/commit/4ada071bfeaacd4cc914da594c06a5a51cbbd8c1
S/O article:
https://stackoverflow.com/questions/76748903/why-is-this-string-blank-after-the-procedure-returns-it-odin

Fixed in d6a2075ad4902d659e23053619e734efc6f453d0


Can the program remove frames of dynamic objects?If so, where?

Can the program remove frames of dynamic objects?If so, where?

Hey @hanbing123-cool, I didn't get this question. If you mean to remove dynamics objects in LIDAR frames, the answer is yes.
As showing in the demo and applications, we provide the script to clean LIDAR frames here: https://github.com/PRBonn/LiDAR-MOS#clean-the-scans

After I have executed "python3 utils/scan_cleaner.py", why do I still have dynamic objects in the frame under "clean_SCANS" when playing with your visualizer?

How can I get a frame that removes a dynamic object?And then send him to the SLAM system

After I have executed "python3 utils/scan_cleaner.py", why do I still have dynamic objects in the frame under "clean_SCANS" when playing with your visualizer?

Could you please show the results you got?

I used the command "python3 utils/visualize_mos.py -d data" +"clean_scans" (including removing dynamic frames) to play back,I thought "python3 utils/scan_cleaner.py" generated the frames with the removal of the motion. I now want to know what folder the frames with the removal of the motion are in.Sorry I haven't looked at the source code yet.

The visualizer we provide so far can only visualize the move object segmentation results, which only show scans frame by frame. You could use scan_cleaner to clean the scan but it then cannot be visualized via visualize_mos, since the number of points is different from the raw point cloud.
We didn't release the visualizer for the clean map so far and we may release it later.

Can you recommend a data set that has label files and IMU and GNSS data? Because the data set you provided does not have IMU information, the data set I downloaded before does not have label file. My SLAM can't be tested

If I use my own data set, I do not have a label file, can I use your program to remove dynamic objects? Use your program to speculate whether there is a label file? Looking forward to your reply.

Can you recommend a data set that has label files and IMU and GNSS data? Because the data set you provided does not have IMU information, the data set I downloaded before does not have label file. My SLAM can't be tested

We will probably release our dataset this year. It will include all the data.

If I use my own data set, I do not have a label file, can I use your program to remove dynamic objects? Use your program to speculate whether there is a label file? Looking forward to your reply.

I don't how different is your data from the KITTI dataset. I therefore can not tell whether it works directly with the pre-trained model or not. You could anyway give it a try and I will appreciate it if you could later also let me know whether it works or not.
Another solution is that we also provide the point cloud labeling tool here. You could also some of your data and fine-tune our model to make it work in your specific environment.

Since there is no further update, I would like to close this issue. If you have any further questions, please feel free to ask me to reopen it.

If I use my own data set, I do not have a label file, can I use your program to remove dynamic objects? Use your program to speculate whether there is a label file? Looking forward to your reply.

您好，请问您成功用这个程序的输出跑通SLAM程序了嘛？可否交流一下呢


Add Llama.cpp Embeddings Support

This PR adds support for using llama.cpp server as a provider of embeddings.
Although llama.cpp server provides an OpenAI compatible endpoint for any embeddings model, tiktoken is often not the right tokeniser so it's not possible to use the OpenAIEmbeddings pathway.
In order to account for how the tokeniser changes based on the model, the implementation uses llama.cpp server's endpoints for tokenisation as opposed to trying to handle it locally - it'll use a lot more API calls, but the benefit of being locally hosted is this shouldn't really matter!

Thanks @Kwigg! Will look in the coming days

@Kwigg could you please:

make a simple example script at examples/docqa/chat-llama-cpp-emdbed.py with basic instructions at the top, that demonstrates the usage of this, or alternatively, add this as an optional setting in examples/docqa/chat.py with basic instructions at the top,
a short doc page at docs/notes/llama-cpp-embeddings.md pointing to this example,
update the nav section in mkdocs.yaml to refer to this doc page

I believe I've made the requested changes, please could you have a look over them?
I also refactored the embedding models.py, as I noticed that only OpenAI embeddings functioned in the examples/docqa/chat.py example with a ChromaDB vector store. Previously, ChromaDB would complain that the function didn't match its expected function signature and throw up errors. This change should fix that!

Thank you @Kwigg  -- fixed a minor conflict and created PR #648 which duplicates this, and also enables running tests. Will merge once passed.

Closing, since dupe #648 was merged.


Export base schema classes

❓ Type of change

[x] 🐞 Bug fix (a non-breaking change that fixes an issue)
[x] 👌 Enhancement (improving an existing functionality like performance)
[ ] ✨ New feature (a non-breaking change that adds functionality)
[ ] ⚠ Breaking change (fix or feature that would cause existing functionality to change)

📚 Description
When a vine validator is exported by another package (e.g. a common package in a monorepo), which is built by tsc it cannot be compiled since tsc cant reference the schema base classes. By exporting the classes and including the path in the package.json exports this gets fixed.
📝 Checklist

[ ] I have linked an issue or discussion.
[ ] I have updated the documentation accordingly.

hey!
could you provide a minimal reproduction repro please?
i would like to see if it can be fixed without exporting these classes
Thanks

hey!
could you provide a minimal reproduction repo please? i would like to see if it can be fixed without exporting these classes
Thanks

Sure, when you try to build the project it will fail since it cannot include the types: https://github.com/adrianNEMO/vine-package-example

Merged in #48


Traditional features migration to Boards and Tasks

Inside Create Notes, the checkbox is not visible where as when displaying the same note, checkboxes are visible.


Update quickstart-linux.md

Update command name from "connection-string show" to "show-connection-string"

@eustacea : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@eustacea In version 0.9.8 of the IoT CLI extension, the command show-connection-string was deprecated in favor of connection-string show: https://github.com/Azure/azure-iot-cli-extension/blob/dev/HISTORY.rst#098
The command will work as written if you update to the latest version of the CLI extension: az extension update --name azure-iot
#please-close


[WIP] lytico/gtksharp/init linux DO NOT MERGE

DO NOT MERGE!
this PR is here to see what i'm working on
and to avoid double work of others and me
it will be replaced by a cleaner one later

A good list.  Since there are no specific tasks set up yet this has me wondering what I can start working on.

@redjaguar it's not ready at all, i've to find out patterns how to implement stuff with minimal effort.
if i got it, i will make a new pull request, with one clean example how to implement a handler.

@lytico Let me know when you want a review or feedback.

this is the Core part of the changes, ready to review:
https://github.com/jsuarezruiz/maui-linux/pull/5


[Swift] cannot unmarshal hex string without 0x prefix

Describe the bug
Platform: iOS
Language: Swift
WalletCore.podspec version = 2.6.5
Using SwiftPackageManager
Testnet: Goerli // chainid: 05
JSON-RPC call: eth_sendRawTransaction
There are conflicting instructions on how the payload should be formed and as a result the JSON-RPC request using EthereumSigningInput fails.
The code example from the WalletCore is:
    func testSignERC20Transfer() {
        let input = EthereumSigningInput.with {
            $0.chainID = Data(hexString: "01")!
            $0.nonce = Data(hexString: "00")!
            // txMode not set, Legacy is the default
            $0.gasPrice = Data(hexString: "09c7652400")! //<PHONE_NUMBER>0
            $0.gasLimit = Data(hexString: "0130B9")! // 78009
            $0.toAddress = "0x6b175474e89094c44da98b954eedeac495271d0f" // DAI
            $0.privateKey = Data(hexString: "0x608dcb1742bb3fb7aec002074e3420e4fab7d00cced79ccdac53ed5b27138151")!
            $0.transaction = EthereumTransaction.with {
                $0.erc20Transfer = EthereumTransaction.ERC20Transfer.with {
                    $0.to = "0x5322b34c88ed0691971bf52a7047448f0f4efc84"
                    $0.amount = Data(hexString: "1bc16d674ec80000")! //<PHONE_NUMBER>000000000
                }
            }
        }
        let output: EthereumSigningOutput = AnySigner.sign(input: input, coin: .ethereum)

        XCTAssertEqual(output.encoded.hexString, "f8aa808509c7652400830130b9946b175474e89094c44da98b954eedeac495271d0f80b844a9059cbb0000000000000000000000005322b34c88ed0691971bf52a7047448f0f4efc840000000000000000000000000000000000000000000000001bc16d674ec8000025a0724c62ad4fbf47346b02de06e603e013f26f26b56fdc0be7ba3d6273401d98cea0032131cae15da7ddcda66963e8bef51ca0d9962bfef0547d3f02597a4a58c931")
        XCTAssertEqual(output.data.hexString, "a9059cbb0000000000000000000000005322b34c88ed0691971bf52a7047448f0f4efc840000000000000000000000000000000000000000000000001bc16d674ec80000")
    }

Whereas on the official website
https://developer.trustwallet.com/wallet-core/integration-guide/wallet-core-usage#multi-coin-wallet
it is:
let input = EthereumSigningInput.with {
    $0.chainID = Data(hexString: "01")!
    $0.gasPrice = Data(hexString: "d693a400")! // decimal<PHONE_NUMBER>
    $0.gasLimit = Data(hexString: "5208")! // decimal 21000
    $0.toAddress = "0xC37054b3b48C3317082E7ba872d7753D13da4986"
    $0.transaction = EthereumTransaction.with {
       $0.transfer = EthereumTransaction.Transfer.with {
           $0.amount = Data(hexString: "0348bca5a16000")!
       }
    }
    $0.privateKey = wallet.getKeyForCoin(coin: .ethereum).data
}

Nonce for example is missing in the web example, so we don't know whether it is optional.
The request using both examples on testnet (either as they are, or using ones own data) fails with the following error:
invalid argument 0: json: cannot unmarshal hex string without 0x prefix into Go value of type hexutil.Bytes
Based on searching for the error message the amount should not have leading zeros, which I have tried and does not work.
Some hexadecimals used in the examples (either on web or in code) use 0x prefix while others do not. This is inconsistent.
TLDR;
Inconsistent web docs and SDK examples.
Ambiguous expected hexadecimal representation.
No examples work when used in a testnet JSON-RPC call
To Reproduce
Attempt to perform a send raw transaction on the Goerli testnet.
https://ethereum.org/en/developers/docs/apis/json-rpc/#eth_sendrawtransaction
Expected behavior
I expected the JSON-RPC request to work using the tests template provide where my own data was replaced.
I expected the documentation to align with the code sample.
Screenshots
/
Additional context
The JSON-RPC calls for getBalance DO work. (since no EthereumSigningInput is used)
The JSON-RPC response is 200.
Based on that I am narrowing the problem down to the EthereumSigningInput parameters or output.

Resolved by prepending "0x" to the output and using 0x for all parameters.
I would still love some cleaning up of the WalletCore and/or website.


Let field tests work with 256-bit curves like secp256k1

Description
When implementing secp256k1, we find that the field tests do not work well with curves whose modulus bit numbers are 64 * K.
This PR fixes so.

Before we can merge this PR, please make sure that all the following items have been
checked off. If any of the checklist items are not applicable, please leave them but
write a little note why.

[x] Targeted PR against correct branch (master)
[x] Linked to GitHub issue with discussion and accepted design OR have an explanation in the PR that describes this work.
[x] Re-reviewed Files changed in the GitHub PR explorer

N/A:

[ ] Wrote unit tests
[ ] Updated relevant documentation in the code
[ ] Added a relevant changelog entry to the Pending section in CHANGELOG.md

I will come back and cross-check on this.

This PR is no longer the solution---instead of allowing an additional limb, we now generalize the code into supporting full limbs. Closed.


Alpha Notes Help

Describe your problem here.
I'm trying to make some notes invisible but it does not work. Really nothing else to say. Here's the code for reference:
function onStepHit()
    if curStep == 2016 then
        doTweenAlpha('byebyenote1', 0, 0, 5, 'linear');
        doTweenAlpha('byebyenote2', 1, 0, 5, 'linear');
        doTweenAlpha('byebyenote3', 2, 0, 5, 'linear');
        doTweenAlpha('byebyenote4', 3, 0, 5, 'linear');
    end
    if curStep == 2320 then
        doTweenAlpha('welcomebacknote1', 0, 1, 0.01, 'linear');
        doTweenAlpha('welcomebacknote2', 1, 1, 0.01, 'linear');
        doTweenAlpha('welcomebacknote3', 2, 1, 0.01, 'linear');
        doTweenAlpha('welcomebacknote4', 3, 1, 0.01, 'linear');
    end
end

Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows x64
Did you edit anything in this build? If so, mention or summarize your changes.
I didn't edit anything that would effect modcharts

notetweenalpha()

NoteTweenAlpha()

How do I use this command?

function onStepHit()
if curStep == 2016  then
noteTweenAlpha("NoteAlpha1", 0, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha2", 1, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha3", 2, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha4", 3, 0, 0.5, circInOut)
end
if curStep == 2320 then
noteTweenAlpha("NoteAlpha1", 0, 1, 0.1, linear)
noteTweenAlpha("NoteAlpha2", 1, 1, 0.1, linear)
noteTweenAlpha("NoteAlpha3", 2, 1, 0.1, linear)
noteTweenAlpha("NoteAlpha4", 3, 1, 0.1, linear)
end
end

function onStepHit()
if curStep == 2016  then
noteTweenAlpha("NoteAlpha1", 0, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha2", 1, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha3", 2, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha4", 3, 0, 0.5, circInOut)
end
if curStep == 2320 then
noteTweenAlpha("NoteAlpha1", 0, 1, 0.1, linear)
noteTweenAlpha("NoteAlpha2", 1, 1, 0.1, linear)
noteTweenAlpha("NoteAlpha3", 2, 1, 0.1, linear)
noteTweenAlpha("NoteAlpha4", 3, 1, 0.1, linear)
end
end

NoteTweenAlpha()

How do I use this command?

noteTweenAlpha("NoteAlpha1", 0, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha2", 1, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha3", 2, 0, 0.5, circInOut)
noteTweenAlpha("NoteAlpha4", 3, 0, 0.5, circInOut)

the correct order is 0,1,2 and 3 of the opponent

Easy! If you want to make it transparent, just make the note texture image completely in PNG (I mean invisible like go to the image, then control + A and press delete)!


Fix a typo

Fixing some simple typos
i -> if
it -> it's
has already executed and executed - > has already executed and finished

ping

/lgtm

/approve
/kind cleanup

Thanks for the PR!


[SPARK-13897][SQL] RelationalGroupedDataset and KeyValueGroupedDataset

What changes were proposed in this pull request?
Previously, Dataset.groupBy returns a GroupedData, and Dataset.groupByKey returns a GroupedDataset. The naming is very similar, and unfortunately does not convey the real differences between the two.
Assume we are grouping by some keys (K). groupByKey is a key-value style group by, in which the schema of the returned dataset is a tuple of just two fields: key and value. groupBy, on the other hand, is a relational style group by, in which the schema of the returned dataset is flattened and contain |K| + |V| fields.
This pull request also removes the experimental tag from RelationalGroupedDataset. It has been with DataFrame since 1.3, and we have enough confidence now to stabilize it.
How was this patch tested?
This is a rename to improve API understandability. Should be covered by all existing tests.

cc @liancheng and @sameeragarwal

Test build #53606 has started for PR 11841 at commit 24eaf42.

Test build #53606 has finished for PR 11841 at commit 24eaf42.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53606/
Test FAILed.

Test build #53607 has started for PR 11841 at commit 3620da9.

LGTM except for one MiMA check question.

Test build #53607 has finished for PR 11841 at commit 3620da9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53607/
Test FAILed.

retest this please

Test build #53617 has started for PR 11841 at commit 3620da9.

Test build #53617 has finished for PR 11841 at commit 3620da9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53617/
Test PASSed.

Merged build finished. Test PASSed.

Thanks - merging in master.


Make dutch auction run for full call time

Might have to change some tests that test:

after margin-call, but before auction starts.

Tests currently use the deployed contract, but I guess we should do a .new() in the tests

Pull Request Test Coverage Report for Build 6911

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 6876:
0.0%

Covered Lines:
1245

Relevant Lines:
1245

💛  - Coveralls


RuntimePermission for saving on Android Version 23+

On my android version (27) runtime permissions are needed to allow saving the pdf to external storage.
In the sample's mainactivity java I wanted to add those but I need to upgrade the min SDK in the gradle and manifest to use functions like ActivityCompat.requestPermissions().
Now I got lost a bit on how exactly to upgrade this project so I can use latest sdk in the sample?
Any advice? Thanks.

Ok after updating SDK and everything in Android Studio saving with runtime permissions worked with following changes for API 24:
build.grade of sample module

compileSdkVersion 24
removing buildToolsVersion
implementation 'com.android.support:appcompat-v7:24+'

And thats all I think. ;)

Closing as this is no longer an issue with the current version


[#160984488] Add CSRF protection

What
We'd like to make sure the application is well protected against the
CSRF. @richardTowers has well spotted that helmet middleware is not
doing that for us, meaning we need to take an action to assure ourselves
we deliver secure software.
The csurf npm package is a middleware that can do things for us
without much effort.
After introduction of csurf package and the implementation, the thests
started to fail due to missing function.
This is a hacky way to assure the function exists while not all of the
middlewares have been setup.
Yet again, this shows lacks in implementation of the tests.
How to review
⚠ I have not tested this, since I don't have my environment running. ⚠

Install dependencies
Code review
Run the application
Navigate around as much as possible to make sure nothing is restricted
Attempt to send few forms, particularly user invite and permission management
Attempt to submit a form from external service (see code sample below for help)

Code sample
Make sure to fill in the URL properly.
<html>
  <body>
    <form action="__PLEASE_PROVIDE_URL_TO_INVITE_PAGE__" method="post">
      Email: <input type="email" name="email"<EMAIL_ADDRESS>      <br>
      <button type="submit">powah!</button>
    </form>
  </body>
</html>

I do not think this is working, it is unable to verify the token. Maybe it is because the problem described in this comment: https://github.com/expressjs/csurf/issues/52#issuecomment-74426892 so that csrf must be done after parsing the bodt.

Cool.
Now have tested (thanks Alex) and can confirm it works as expected.

It fails when submitting the aggrements.
To reproduce:
echo '{"content": "# Foo\n\nblablablaa\n"}' > agreement.json

DEPLOY_ENV=alext
ACCOUNTS_PASSWORD="$(aws s3 cp s3://gds-paas-${DEPLOY_ENV}-state/cf-vars-store.yml  - | awk '/secrets_paas_accounts_admin_password/ {print $2}')"
ACCOUNTS_URL=https://accounts.${DEPLOY_ENV}.dev.cloudpipeline.digital
curl -u admin:${ACCOUNTS_PASSWORD}     -H "Content-Type: application/json"     -X PUT -d @./agreement.json     ${ACCOUNTS_URL}/documents/TnC

Then visit the admin app and click on accept.
It fails with: ForbiddenError: invalid csrf token

This makes that work:
$ git diff
diff --git a/src/components/terms/middleware.ts b/src/components/terms/middleware.ts
index 3362a81..59eee11 100644
--- a/src/components/terms/middleware.ts
+++ b/src/components/terms/middleware.ts
@@ -22,7 +22,10 @@ export function termsCheckerMiddleware(config: IAccountsClientConfig): express.H

   app.get('/agreements/:name', sync(async (req, res) => {
     const document = await accounts.getDocument(req.params.name);
-    res.send(termsTemplate.render({document}));
+    res.send(termsTemplate.render({
+      document: document,
+      csrf: req.csrfToken(),
+    }));
   }));

   app.post('/agreements', sync(async (req, res) => {

But maybe we should test it and add some sort of test to check that the terms can be accepted at least.
Also, maybe you want to refactor the terms part to include inside of the router.ts :thinking:

Good spot!
I honestly don't know why is the terms module so much separated from the rest of the app... In fact that's the reason I didn't initially spot that handler.
I think it's to big of a change for this PR.

I am sorry being a PITA, but test don't pass... :man_shrugging:

This can make the test pass:
$ git diff
diff --git a/src/components/terms/middleware.test.ts b/src/components/terms/middleware.test.ts
index be7158c..0a6ef97 100644
--- a/src/components/terms/middleware.test.ts
+++ b/src/components/terms/middleware.test.ts
@@ -26,6 +26,7 @@ app.use((req: any, _res: any, next: any) => {
     user_id: fakeUserID,
   }, tokenKeys[0]);
   req.token = new Token(accessToken, tokenKeys);
+  req.csrfToken = function csrfToken () { return "foo" }
   next();
 });

This would be the simplest, otherwise you need to refactor the test to get the csrf token in the post.
But we are, again, ignoring and working around the feature in the tests instead of testing it.  Although this is what has been done in other tests.

The ideal approach is refactoring this module to our own router style... This would then be testing the csurf library along with it... It was a real nightmare to make it work with the library at current state, so I decided against it and did something similar to your suggestion...
We could be mocking the check if that makes you feel any better?


 If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.

Probably the previous Let's Encrypt root cross-signing cert expiration. Is https://superuser.com/questions/1679204/curl-on-ubuntu-14-all-lets-encrypt-certificates-are-expired-error-60 helpful at all?

Same problem.

Theres also this:
Android made a design decision to ignore the expiry on root certificates (it's arguable that expiry on root certificates doesn't make much sense in the first place).
"lets encrypt" is a relatively new CA and to support existing systems their root was "cross-signed"* by DST. The DST root certificate has now expired but because of the aforementioned android behavior a cross signature is still useful for supporting clients running old versions of andriod.
When such a chain is used on a system with a modern root certificate list the cross signature should be ignored and the IRSG (lets encrypt) root should be used. Unfortunately openssl 1.0.x does not handle this scenario correctly.
This issue can be worked around by removing the old DST root certificate. When this is done the chain will correctly be built to the IRSG root.

My understanding is that technically the "cross signature" consists of an "intermediate certificate" with the same content and key as Lets Encrypt's root certificate.


fix: Constrain to known-to-work versions of commitizen

Related:

https://github.com/commitizen-tools/commitizen/issues/1024

i am suffering from this issue. please merge this pr quickly:)

@akfmdl the problem this was trying to address was quickly addressed by the commitizen folks in  https://github.com/commitizen-tools/commitizen/pull/1026 and shipped in 3.18.4.


cilium-agent crashes with 'protocol not supported'

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Trying to add a new ARM64 (Raspberry Pi 4B) node to an existing cluster.
I was expecting the cilium POD will start normally, however it was in CrashLoopBackOff status.
It looks like there is a 'protocol not supported' error raised.
Cilium Version
cilium-cli: 0.12.0 compiled with go1.18.4 on darwin/arm64
cilium image (default): v1.12.0
cilium image (stable): v1.12.0
cilium image (running): v1.12.0
Kernel Version
Linux worker-arm64-001 5.15.0-1012-raspi #14-Ubuntu SMP PREEMPT Fri Jun 24 13:10:28 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Kubernetes Version
All nodes are on 1.24.3-00
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
Sysdump
cilium-sysdump-20220801-214249.zip
Relevant log output
level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=<IP_ADDRESS>/16
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://x.x.x.x:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=debug msg="Starting new controller" name=k8s-heartbeat subsys=controller uuid=a728903d-778d-493e-af14-7436652e2ee5
level=debug msg="Controller func execution time: 4.112µs" name=k8s-heartbeat subsys=controller uuid=a728903d-778d-493e-af14-7436652e2ee5
level=debug msg="Skipping Leases support fallback discovery" subsys=k8s
level=debug msg="Starting new controller" name=endpoint-gc subsys=controller uuid=91f98097-0846-4831-aae6-fcae82e7a9b0
level=debug msg="Controller func execution time: 6.076µs" name=endpoint-gc subsys=controller uuid=91f98097-0846-4831-aae6-fcae82e7a9b0
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"bpf-lb-sock\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=<IP_ADDRESS> mtu=1500 subsys=mtu
level=debug msg="Performing regular background work" subsys=nodemanager syncInterval=1m0s
level=fatal msg="Error while creating daemon" error="protocol not supported" subsys=daemon

Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Are you installing with helm?

I think my problem might be RTFM.
Kernel modules were missing, which are in a different package on Ubuntu (kernel-modules-extra): xfrm_user/xfrm_interface
I might still have other issues with datapath-loader but I will rebuild the node from scratch first.

How does one get the required modules on arm64 (Raspberry Pi running Ubuntu 22.04)?
I'm using Helm and that error should say which protocol it's talking about.
level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=<IP_ADDRESS>/16
level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a0a:0:0:0/96"
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://<IP_ADDRESS>:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"bpf-lb-sock\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=<IP_ADDRESS> mtu=1500 subsys=mtu
level=fatal msg="Error while creating daemon" error="protocol not supported" subsys=daemon

How does one get the required modules on arm64 (Raspberry Pi running Ubuntu 22.04)? I'm using Helm and that error should say which protocol it's talking about.
level=info msg="Auto-disabling \"enable-bpf-clock-probe\" feature since KERNEL_HZ cannot be determined" error="Cannot probe CONFIG_HZ" subsys=daemon
level=info msg="Using autogenerated IPv4 allocation range" subsys=node v4Prefix=<IP_ADDRESS>/16
level=info msg="Using autogenerated IPv6 allocation range" subsys=node v6Prefix="f00d::a0a:0:0:0/96"
level=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="https://<IP_ADDRESS>:6443" subsys=k8s
level=info msg="Connected to apiserver" subsys=k8s
level=info msg="Trying to auto-enable \"enable-node-port\", \"enable-external-ips\", \"bpf-lb-sock\", \"enable-host-port\", \"enable-session-affinity\" features" subsys=daemon
level=info msg="Inheriting MTU from external network interface" device=eth0 ipAddr=<IP_ADDRESS> mtu=1500 subsys=mtu
level=fatal msg="Error while creating daemon" error="protocol not supported" subsys=daemon

Please make sure you have linux-modules-extra-raspi  package installed. After you installed it, I recommend to reboot your node and it should come up normally.


feat: support canisters and principals in dfx ledger account-id

This change adds new capabilities to the dfx ledger account-id
command:

The --subaccount option allows you to get an account id for a
subaccount other than the default one.

The --of-principal option allows you to specify the controller
of the subaccount to compute addresses that your identity doesn't
control.

The --of-canister option allows you to specify canister alias
instead of the principal.

Examples:
$ dfx ledger account-id --of-principal 53zcu-tiaaa-aaaaa-qaaba-cai
e9b0cdfcb858f5538d0d1d8d0a2700ccda299451c88bf686d00a3d8fb74dcfea

$ dfx ledger --network small02 account-id --of-canister ledger_demo
e9b0cdfcb858f5538d0d1d8d0a2700ccda299451c88bf686d00a3d8fb74dcfea

$ dfx ledger account-id \
  --of-principal 53zcu-tiaaa-aaaaa-qaaba-cai \
  --subaccount<PHONE_NUMBER>000000000000000000000000000000000000000000000000000001
adddf401b1936409217493ed02b144ede539c6397c270b4ca8069f9930a4f4c9

This change also improves the error reporting by displaying error
causes if there are any. Example:
$ dfx ledger account-id \
  --of-principal 53zcu-tiaaa-aaaaa-qaaba-cai \
  --subaccount<PHONE_NUMBER>0000000000000000000000000000000000000000000000000000000
Error: Subaccount '00000000000000000000000000000000000000000000000000000000000000000' is not a valid hex string
Caused by:
  Odd number of digits

Fixes #1981.

Would you please add a note to the changelog?

Good point, done!


[C++] Simplify IPC writer for dense unions

In ARROW-10580 we fixed the Arrow C++ implementation so that dense union offsets are always (non-strictly) monotonic for a given child, as mandated by the spec.
The IPC writer implementation, however, still assumes that dense union offsets may be in any order:
https://github.com/apache/arrow/blob/5719576c611929dd790f7f8a1ae3169a8f96f7f1/cpp/src/arrow/ipc/writer.cc#L476-L485
This can probably be simplified, making it slightly less costly to emit a sliced union array.
Reporter: Antoine Pitrou / @pitrou
Related issues:

[C++] When Validating, ensure DenseUnionArray offsets are increasing (relates to)

Note: This issue was originally created as ARROW-17339. Please see the migration documentation for further details.

Hi, I'd like to work on this if it is still available. Thanks.


Configuration into init.vim

How can I configure in the init.vim? I configured like that ant it works:
init.vim:
call plug#begin('~/.vim/plugged')

Plug 'Mofiqul/vscode.nvim'

call plug#end()

colorscheme vscode

But, I dont know how can I change configurations of the theme.

I think this will help you.
https://www.reddit.com/r/neovim/comments/i5y18f/how_can_i_source_lua_from_initvim/


Deprecated API WARNing in v<IP_ADDRESS>

Couldn't find this issue already raised in dropwizard-guice, so thought I'd ask about it.
When starting up my dropwizard app with dropwizard-guice I get the following warning.
WARN  [2016-11-15 20:49:44,414] com.google.inject.servlet.InternalServletModule: You are attempting to use a deprecated API (specifically, attempting to @Inject ServletContext inside an eagerly created singleton. While we allow this for backwards compatibility, be warned that this MAY have unexpected behavior if you have more than one injector (with ServletModule) running in the same JVM. Please consult the Guice Documentation at https://github.com/google/guice/wiki/Servlets for more information.

For reference - it was fixed in guicier https://github.com/HubSpot/dropwizard-guicier/issues/11

Fixed by #102 thanks @ikatkov

Would it be possible to release dropwizard-guice-<IP_ADDRESS> now?

Yup, sorry for the delay just released now

Is there a link to build process? When should I expect to see it in maven?
On Wed, Feb 1, 2017 at 11:16 AM, Jonathan Haber<EMAIL_ADDRESS>wrote:

Yup, sorry for the delay just released now
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/HubSpot/dropwizard-guice/issues/98#issuecomment-276752425,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABJ7z1wbKArbDP-97hs8EAydDdKokhUnks5rYNocgaJpZM4KzB_4
.

Ugh my fault, it got pushed to Sonatype staging but I never released it. Just took care of that now, it should show up in Maven central soon


Allow to select previously create virtual sensors in group

Todo:

[ ] Translation keys
[ ] Move selector higher in the UI

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

27 of 27 (100.0%)  changed or added relevant lines in 3 files are covered.
11 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.4%) to 92.706%

Files with Coverage Reduction
New Missed Lines
%

custom_components/powercalc/sensors/group.py
11
93.75%

Totals

Change from base Build<PHONE_NUMBER>:
-0.4%

Covered Lines:
2237

Relevant Lines:
2413

💛  - Coveralls


New single installer .exe switches

What are the new single installer .exe switches so we can use in command line to perform silent installation or select the language to more automate the installation?

Thanks for your comment, @rhakabi. I've assigned this issue for review. Are you referring to this article: https://docs.microsoft.com/en-us/power-bi/desktop-get-the-desktop?

Yes, how to perform a silent install with the new installer?
Get Outlook for Androidhttps://aka.ms/ghei36

From: Michael Blythe<EMAIL_ADDRESS>Sent: Friday, July 26, 2019 4:45:15 PM
To: MicrosoftDocs/powerbi-docs<EMAIL_ADDRESS>Cc: Roger Halabi<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [MicrosoftDocs/powerbi-docs] New single installer .exe switches (#1273)
Thanks for your comment, @rhakabi. I've assigned this issue for review. Are you referring to this article: https://docs.microsoft.com/en-us/power-bi/desktop-get-the-desktophttps://urldefense.proofpoint.com/v2/url?u=https-3A__docs.microsoft.com_en-2Dus_power-2Dbi_desktop-2Dget-2Dthe-2Ddesktop&d=DwMCaQ&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=YNIvcxiPBC2_DkMsMA3Qb29Yb-Q3a-QzZgoNqqJoGhk&m=2rllSdn5Ha2PbMjwwYy6NI2uwZARZMDg8IYYjpVZ-RU&s=ANx77bpBFxP-C6J5LVXF1pAN6UEDYQ7qv743-sg1DAw&e=?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MicrosoftDocs_powerbi-2Ddocs_issues_1273-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMW6YIHKO3GFHIBL6E3UQNDQBNV6XA5CNFSM4IHDMA3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25ZBKI-23issuecomment-2D515608745&d=DwMCaQ&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=YNIvcxiPBC2_DkMsMA3Qb29Yb-Q3a-QzZgoNqqJoGhk&m=2rllSdn5Ha2PbMjwwYy6NI2uwZARZMDg8IYYjpVZ-RU&s=IgkKy4Z7rN6fArpcNpKUkTPQ7Hdfj03bfdmQOXaFHcA&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMW6YIEEY4OLTQIPLVBVZQLQBNV6XANCNFSM4IHDMA3A&d=DwMCaQ&c=p0oa49nxxGtbbM2qgM-GB4r4m9OlGg-sEp8sXylY2aQ&r=YNIvcxiPBC2_DkMsMA3Qb29Yb-Q3a-QzZgoNqqJoGhk&m=2rllSdn5Ha2PbMjwwYy6NI2uwZARZMDg8IYYjpVZ-RU&s=d6wsBOywf7OWhNw2hshaG1c_gTIX_SpfIbrWG0oYyyM&e=.


re #1034 Data encryption blog post

Changes:

Added a blog post on the encryption of data that apiman stores.
Added myself as an author, and re-ordered list of names alphabetically (by first name).

--
Screenshots:

--
JIRA: https://issues.jboss.org/browse/APIMAN-1034
cc @EricWittmann @msavy - Sorry that this took forever, I wanted to make sure I was adding enough detail. A few comments:

Please check the accuracy of what I wrote, especially the comment about pre- and post-marshalling when storing in Elasticsearch toward the end. I was basing this off of my notes from a bootstrapping meeting and by looking at the code.
I didn't get too detailed for things like setting up the structure for a plugin, but did provide references. Maybe should have done that for generating an encrypted key as well.
I'd really like to add comments in today, if possible. It's a quick task that I think is pretty important. That's one of the reasons I mention "please leave a comment below" at the end of the post.
Last thing is, please don't merge this in yet as I've yet to commit the built files. This is just for you guys to read, review, and provide feedback (if you have the time!). It would be nice if this were done automatically in Travis CI / Jenkins.

Thanks!

Blob is here, but obviously looks better when rendered locally (see screenshot).

@EricWittmann @msavy - Made suggested changes, updated the date, and added Disqus commenting. See updated screenshot below.
It is desperately missing images, but I didn't think any would be helpful so decided to omit them completely. Once you guys are okay with it, let me know and I'll check in the built Jekyll files.
--


Collapsible functions

appreciate your patience on this @unholypanda! 🙇 ski season stoke surpassed the dev itch for a little bit there :)
getting a fresh start on this pr now with the new contrib guidelines and pr template
Motivation

more complex contracts such as seaport looked a little cluttered once imported

Solution

Add a +/- collapse toggle to each function in a contract's ABI. This has the effect of making navigation to a specific method easier for complex ABIs, while sacrificing a little friction for contracts with simpler signatures.

Additional Notes

I saw CI will still need to be approved to run, but i've gone through the steps noted in the contributing doc and it all looks green now

This looks awesome! Thanks for taking the time to open another PR after all of the contribution guidelines and GitHub Actions were included. Your patience is much appreciated!
I have some small features that I want to include in follow-up PRs that are related to the collapsible functions here. My plan right now is to merge this tomorrow evening.


Swap EVAC_WAIT_TIME to waitTime in _evac.gnut

Use the variable waitTime instead of the const EVAC_WAIT_TIME,

yeah its just a small slip-up, this PR will prevent players from seeing 18 seconds timer on the dropship even if a custom waitTime is set.

PR speedrun any%

PR speedrun any%

real


[Bug]: Llama 3 answers starting with <|start_header_id|>assistant<|end_header_id|>

Your current environment

The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.29.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.1.85+-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA L4
Nvidia driver version: 550.54.15
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               32
On-line CPU(s) list:                  0-31
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                           6
Model:                                85
Thread(s) per core:                   2
Core(s) per socket:                   16
Socket(s):                            1
Stepping:                             7
BogoMIPS:                             4400.43
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            512 KiB (16 instances)
L1i cache:                            512 KiB (16 instances)
L2 cache:                             16 MiB (16 instances)
L3 cache:                             38.5 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-31
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==<IP_ADDRESS>
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==<IP_ADDRESS>
[pip3] nvidia-cufft-cu12==<IP_ADDRESS>
[pip3] nvidia-curand-cu12==<IP_ADDRESS>
[pip3] nvidia-cusolver-cu12==<IP_ADDRESS>
[pip3] nvidia-cusparse-cu12==<IP_ADDRESS>
[pip3] nvidia-ml-py==12.555.43
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] torch==2.3.0
[pip3] transformers==4.40.2
[pip3] triton==2.3.0
[pip3] vllm-nccl-cu12==<IP_ADDRESS>.4.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-31	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

🐛 Describe the bug
I'm running vLLM via kserve using the following yaml:

YAML file
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: huggingface-nous-research-llama
spec:
  predictor:
    serviceAccountName: "default-editor"
    tolerations:
      - key: "l4-gpu"
        operator: Equal
        value: "present"
      - key: "nvidia.com/gpu"
        operator: Equal
        value: "present"
    model:
      modelFormat:
        name: huggingface
      args:
        - --model_name=llama3
        - --model_id=NousResearch/Meta-Llama-3-8B-Instruct
      resources:
        limits:
          cpu: "26"
          memory: 40Gi
          nvidia.com/gpu: "1"
        requests:
          cpu: "26"
          memory: 40Gi
          nvidia.com/gpu: "1"

I've also tried other sources of Llama 3, including the original from Meta.
Whenever I send a request to vLLM, the response content starts with <|start_header_id|>assistant<|end_header_id|>. I am pretty sure this is because the tokenizer is not adding them to the prompt and the model does it itself.
I have tried setting "add_generation_prompt": True in my requests (which should be the default anyway), and even copying the tokenizer chat template to chat_template, but the result is the same.
It is surprising that I couldn't find any other reports of this behavior with vLLM, so maybe there is something wrong with my setup.

i expected skip_special_tokens=True as a sampling_arg to do the same. Can't get the model to output text not statrting with headers


add formatting requirement to squlalchemy uri option

Purpose

Quick fix to add another line to the FIDESOPS__DATABASE__SQLALCHEMY_DATABASE_URI config option to provide a sample connection string.

Checklist

[x] Applicable documentation updated (guides, quickstart, postman collections, tutorial, fidesdemo, database diagram.

Ticket
Fixes #502

Oh, forgot 1 thing, @conceptualshark do you think we should add to the ### Docs section of Changelog.md?

I wasn't sure if small updates like this should be included; the config option is already in the docs, this is just an actual example.


Release notes for v0.1.0-alpha.24

This is the long-running PR for release notes to be included in the next release.
Community contributors are welcome to comment below with your notes, or open a PR to merge into this branch.
Coders with write access are welcome to add commits on top. Please don't force push, though: it would be sad to accidentally blow away someone else's notes.

Yikes, looks like master is broken!

Yikes, looks like master is broken!

Wait, what?

Yikes, looks like master is broken!

Wait, what?

That commit is 24 days ago, 14fab95 is from 3 days ago.
The breakage is presumably from pulling in a dependency that was released sometime between 3-24 days ago.

But that commit isn't on master. It's in the release-notes branch, which I assume will be all squashed and picked into one commit onto master at the point where we cut a new release, no?

But that commit isn't on master. It's in the release-notes branch, which I assume will be all squashed, rebased and picked into one commit onto master at the point where we cut a new release, no?

Why would a squash/rebase affect any of those lint failures?

Why would a squash/rebase affect any of those lint failures?

Because those do not exist on latest master (e.g. here's a PR that is just several hours old, builds on latest master and is green https://github.com/ethereum/trinity/pull/557). So, once you rebase this branch on latest master, the linting issues should just vaporize.
Or to put it another way, this PR, the release-notes branch does only modify the trinity.rst file and we should feel free to ignore any CI fails in this branch that aren't directly related to changes to the trinity.rst file.
When the time comes, we'll squash all changes into one commit, rebase on top of latest master and all issues should magically disappear under the assumption that master is passing which seems to be the case.
The alternative would be rebase this branch now and then force push it but the issue description says this should be avoided.

Please don't force push, though: it would be sad to accidentally blow away someone else's notes.

Because those do not exist on latest master (e.g. here's a PR that is just several hours old, builds on latest master and is green #557). So, once you rebase this branch on latest master, the linting issues should just magically vaporize (hint: because latest master is on a newer mypy version etc).

Hm, but every new test run is supposed to merge with master before running tests, so it should have all the latest changes on master.
It's not so much that I want to fix the problem of this PR having a :x: -- it's that this ^ mechanism I thought should prevent the problem of testing on old PRs. If it's not solving the problem, I want to understand why and maybe try to fix it.

I pushed my changes from the last few weeks.

For PR #608 (not yet merged):
- `#608 <https://github.com/ethereum/trinity/pull/608>`_: Tests: Run tests for Constantinople and Petersburg

For PR #623 (not yet merged):
- `#623 <https://github.com/ethereum/trinity/pull/623>`_: Tests: Optimise for faster run times

I'm starting to clean and finish up the release notes for today's trinity release. If you have any more additions, please make a comment instead of editing the PR.

Wow, we clearly waited too long between releases. (I mean we already knew that, but this changelog is enormous.)


querySampleType requiring write permissions

Here is my prompt for permission code
window.plugins.healthkit.requestAuthorization({ 'readTypes'  : ['HKQuantityTypeIdentifierStepCount','HKQuantityTypeIdentifierActiveEnergyBurned' ]}, auth_callback, auth_callback);
And my query code:
window.plugins.healthkit.querySampleType({'startDate': start, 'endDate': finish, 'sampleType': 'HKQuantityTypeIdentifierStepCount'}
The authorisation dialogue appears with 2 toggles as expected initially, however the query code forces the authorisation dialogue again, with the 2 toggles maintaining their "on" stage, but also displays a prompt for write permission for the Identifier being requested (HKQuantityTypeIdentifierStepCount). I do not need write permission and do not explicitly request it in my code. The code I've shown is all the code my application contains that relates to this plugin.
Am I missing something obvious or is this unexpected behaviour?

Hi Farid,
Good point.
The plugin requests read and write permission because of a (in my opinion) flaw in the HealthKit SDK. If say you first request permission to read weight it will show the permission toggle for weight. Later if you ask write permission it will ONLY show the write toggle and you loose the read permission. If later you want to read again it asks for permission. Really crappy behavior imo.
To circumvent this and keep the API simply I've decided to always ask read and write permissions if you need at least one of those. I can however see why this could be suboptimal.
I think I'll add a requestReadPermission and requireWritePermission boolean to every API call, defaulting to true to remain backward compatible.
In the meanwhile you can use the requestAuthorization function before using the querySampleType. Just pass in only the correct read type and you should be able to use it like this (I have not tested this passing an empty writeTypes param, but I should work):
function requestAuthorization() {
    window.plugins.healthkit.requestAuthorization(
        {
          // comma-separated type lists
          'readTypes'  : ['HKQuantityTypeIdentifierStepCount'],
          'writeTypes' : []
        },
        callback, // this is where you can use querySampleType
        callback
    );
  }

Let me know how this works out or you.
Thanks,
Eddy

I've tried both omitting and including an empty array for writeTypes, the behaviour seems identical (permission prompt).
I do actually have the requestAuthorization request on its own in one place (turning on HK integration), and also request authorization before querying the data.
I would be happy to look into this issue if you could give me some pointers regarding the codebase.

Hi, just checked. You'll get the permission prompt only for the read types if you do it like the example above (you can even leave out the writeTypes entirely).
If you're seeing a permission prompt for more options then please try deleting your app first as those permissions tend to accumulate (it shows any previously shown permission as well).

BAM! Then it he me. I was reading your bugreport wrong. You're saying that whatever read permission you ask upfront, the querySampleType function will always ask write permissions as well. That's a bug I will address.

Hi, version 0.3.0 is now available and should fix your trouble.
There's also the new requestReadPermission and requestWritePermission params for a few other methods, see the docs.

Thank you Eddy, the updated version fixes the issue and I'm very pleased with how it's all working now.
If you ever need contributors to this plugin or another one please let me know, I'm keen to give back if I can help in any way :smile:

Great to hear! :+1:
And thanks for offer, I'll keep it in mind ;)

Eddy, I am new to healthkit plugin. Trying to req the authorization like
window.plugins.healthkit.requestAuthorization({
"readTypes": ["HKCorrelationTypeIdentifierBloodPressure"], "writeTypes": []
},
function (value) {
_trace("Access to HKCorrelationTypeIdentifierBloodPressure allowed", true);
},
function (value) {
_trace("Access to HKCorrelationTypeIdentifierBloodPressure rejected", true);
});
in the phonegap application.
The app does NOT display any dialogue but instead crashes 5-6 seconds later.
iOS8, the healthkit is available, 'healthkit' permission is on for the app in the Apple's settings.
Any hint on what I might be doing wrong ?
Thank you.


Feature(ConnectorSdk): Multithreading Guidelines for Accelo Example

Closes https://fivetran.height.app/T-809939
Multithreading Guidelines. Refer comment.

Hi @shivangisingh172 following up for you're review.