WIP: Added function to parse a sample-sheet into dataframes.

Extended _parse_header() to parse all sections of a sample-sheet into dataframes and return them to the caller. This will make it easier and more robust to modify values like 'Lane' on-demand. This is part of an effort to potentially replace the third-party sample_sheet module and parsing method. Lane is a field that is currently read-only once read into the base SampleSheet() object. A robust method to change the values of an existing column or add a column if it's not present is needed.
This is a work-in-progress. I need a more robust method to overwrite the values for Lane in a sample-sheet to support the tellseq code and the plugin refactoring.

tests are still needed.

extract_data_from_sheet() can be removed, as we don't need it anymore.
will wait until Amanda's PR is merged before resolving this PR, as some of the changes overlap e.g. TellSeq SampleSheet() implementations. I don't need to keep mine as they are barebones.
Will want to keep and merge the DFSheet() and set_lane_in_sample_sheet() code.
Tests will be needed for this code.

Amanda's latest merged into master and duplicate code in this PR removed. Flake8 needed.

@AmandaBirmingham I merged your branch in and made the changes to my own branch to have it pass all tests. I think Daniel's comments do have merit and so I was thinking it would be good to focus on whether the new functionality here should be extended or changed and the sheets we will need for end-to-end testing and whatever else we need.

I'm closing this without merging. If we want to use DFSheet() in the future it will be here.


Integration with Graphene

Motivation & problem statement
As #502 questioned that the integration with a LibOS would be interesting feature for a FaaS platform because allowing to run regular app. directly in enclave along with LibOS handling most syscalls may further improve the usability and simplify the usage model. Theoretically, the performance might be improved slightly as well. from the other side, the app. dev. is agnostic to the LibOS/Kernel. so the app. might contains IO/Mem intensive operations that will have negative impact on performance even stability.
Proposed solution
Regarding Graphene, which is a LibOS that can support unmodified app. to run inside enclave, its multiple process feature may not be very useful for FaaS.  In this stage, I think we may need to define an interface for LibOS kind integration first to manage the lifecycle&IO of trusted LibOS enclave instance.

Graphene

Hi @bigdata-memory, thanks for the proposal.
Teaclave is a FaaS-like platform consisting of multiple services. One can register and invoke functions to the platform through the frontend service. Then, Teaclave will help to do authentication, authorization, task preparation, etc and finally dispatch a task to the execution service. Currently, we have three executors for different languages and scenarios. For example, the builtin executor only execute functions written in Rust natively, while, the mesapy executor can run functions written in Python dynamically.
Therefore, to integrate with Graphene, or any other LibOS kind of system in Teaclave, we need to introduce another kind of executor. Luckly, the execution service in Teaclave are designed as a stateless and isolated service which takes a task, executes it and write back the result. Inputs/Outputs of a functions are defined as function arguments, input/output files stored in protected file system. In our design, we prohibit any untrusted I/O  interfaces  (e.g., write date to the untrusted file system, networks I/O) in the executor to protecting sensitive data operated by the function.
Above are some background about our design rationals. Here, I list some concepts involved in integrating with Graphene.

RPC: All Teaclave services communicate with each other via trusted channel (i.e., attested TLS). You can find all protocol definitions in Protobuf in this directory: https://github.com/apache/incubator-teaclave/tree/master/services/proto/src/proto. An execution service instance will periodically fetch tasks from the scheduler service and execute it.
Attestation: We do mutual attestation when establishing trusted RPC channel through attested TLS.
Executor Runtime: Providing interfaces to an executor to read/write input/output files stored in the protected file system.
TeaclaveExecutor Trait: The TeaclaveExecutor trait defines interfaces of a task.

Recently, we added a WebAssembly executor. You can see recents changes to see how to add an executor in Teaclave. Here is a simple document https://teaclave.apache.org/docs/adding-executors/.
The WebAssembly executor is just a reference. For integrating with Graphene (or other LibOS executors), the design may be different. I'm not sure whether to link the LibOS within the execution service or put it as a separate enclave along with the execution service. The later may need additional attestation mechanism.
At last, can you propose a simple design so that we can work on it together? Please let me know if you have any questions regarding to the current Teaclave design. Thanks!

Hi @mssun, Thanks you very much for the detailed explanation on leveraging LibOS kind system in Teaclave.
It is very helpful for us to work on this effectively. the GSGX so far only has relative stable CLI interface and its internal APIs are not designed for 3rd party integration since it's API still under evolving. roughly, we may consider to running GSGX instance as sidecars along with its execution driver service and do mutual attestation with its driver beforehand. It is just my preliminary idea for now, we may define assumption, constraints and requirements for this soon later. Thanks!

In the attestation
page,  the way to binding is We make the certificate cryptographically bound to a specific enclave instance by adding the public key of the certificate in the attestation report., how the service certification can get revoked and re-issued at runtime ? Thanks.

how the service certification can get revoked and re-issued at runtime?

We don't have the revoking mechanism currently.
Right now, there is a freshness timer for the attestation/certificate. Once it's timeout, the service will do another attestation and update the report, re-issue the certificate as well.

I'm also interested in incorporating Graphene as an Executor of Teaclave. However, I think we must take serious consideration about the interfaces of Graphene (mainly system calls).
As @bigdata-memory mentioned, many syscalls are handled by libOS, but there're also many syscalls which are forwarded, either in unaltered way or with some changes, to the OS kernel. These forwarded syscalls includes I/O operations on file system, Networks, as well as timing system (e.g. nanosleep) and synchronization (e.g. futex). Besides, Graphene can also handle interrupts/exceptions.
To avoid using such untrusted interfaces and information leakage on these interfaces, Teaclave implements no untrusted interface. And Graphene aims to support trusted binary, whereas Teaclave assumes potentially malicious function. Such difference in threat model results in the differences on supported/exposed interfaces to binaries/functions running inside Graphene/Teaclave. If we want to leverage Graphene's power, we may need to think about several questions:

Tradeoff between binary compatibility and security: how to deal with untrusted interfaces?
How to run untrusted binary in Graphene securely? Personaly I believe auditing can be a solution
How can Teaclave and Graphene communicate securely? AFAIK, Graphene currently lacks a secure channel implementation.

Hi @ya0guang, Glad to know you also got interested in this. Yes. as you mentioned, the Graphene forwards many syscalls to untrusted host, more specifically, 41 SGX OCALLs to untrusted PAL layer of Graphene and do some kinds of security check. it is similar to have some unsafe calls to external libraries in Rust. we need to be very careful to design a mechanism to audit it and filter out unexpected behaviors according to some terms that can be created for untrusted Pal as a new party.

Hi @mssun, as we talked, the Graphene will download several dependent 3rd party's codebases during building time. e.g.  glibc, mbedtls, uthash, toml, gcc. all downloads are verified by hashcodes, however, the document mentioned will only consume packages from this designated repository and will not download any code from external package registry in page THIRD-PARTY DEPENDENCY VENDORING. does it comply with what is mentioned in the document if linking Graphene as one of submodules in Teaclave project ? Thanks.

In addition, Graphene project licensed as GPL v3.0 along with an addendum, but it is not permissive compatible with Apache 2, the page GPL-compatibility says However, GPLv3 software cannot be included in Apache projects, as talked, looks Graphene cannot be fully integrated with Teaclave at building time, so before Graphene executor is available to use, it needs to be deployed on platform separately, please advice, thanks.

Hi @ya0guang I'm studying the PR #504 as a guide to adding Graphene as an executor. In the example script wasm_simple_add.py, the payload_file: wasm_simple_add_payload/simple_add.wasm is loaded and registered in front service. It looks the payload is not part of measurement of enclave, do I understand it correctly ? Thanks.

Hi @bigdata-memory , I think you're right about this example! Since the enclave measurement is determined by its initial code and data before instantiation, the payload which is loaded into Teaclave after instantiation cannot affect the measurement. At least for MesaPy and WASM executor, the payload was uploaded by the users and then loaded after enclave instantiation. However I guess the story may be different for the Builtin executor, because if you want to add a function, you may need to change the executor itself directly first before an enclave is generated.

In addition, Graphene project licensed as GPL v3.0 along with an addendum, but it is not permissive compatible with Apache 2, the page GPL-compatibility says However, GPLv3 software cannot be included in Apache projects, as talked, looks Graphene cannot be fully integrated with Teaclave at building time, so before Graphene executor is available to use, it needs to be deployed on platform separately, please advice, thanks.

@bigdata-memory, thanks for pointing out this issue. I just checked that Graphene is in LGPL v3.0 license.
As stated in ASF 3rd Party License Policy:

You may NOT include the following licenses within Apache products:

Places restrictions on larger works: GNU LGPL 2, 2.1, 3

Therefore, I suggest NOT to include any Graphene code in Teaclave or as a third-party dependency.
Given this situation, if we plan to integrate with Graphene. I suggest to do the followings in Teaclave:

Provide interfaces for any libOS executors
Write documents on integrating with Graphene
Implement a mock libOS implementation for testing only

@mssun, thank you for the suggestion. regarding Provide interfaces for any libOS executors, the 3rd-party libOS would be running in a separate process/enclave space, should we keep it running or once per each request ? thanks.

We drafted the design of GSGX enabling on Teaclave, Please review. thanks.

teaclave_libos_gsgx.pdf

@bigdata-memory Regarding the design, how is that going to facilitate the attestation flow in the Teaclave? Will it provide additional features or benefits on top of it?

@hanboa Thank you for this question, Yes, you are right, the attestation flow is not explicitly shown here, In this design draft, we consider to retrieve the quote from the component Teaclave premain, honestly, it is not a graceful way to fetch it through this customized component, I proposed one solution to GSGX community, but their maintainer disagreed that it is a legitimate use to support the feature of writing the quote out to /var/run/ directory. However, the component GrapheneDriver is designed handle it anyway. In GrapheneController, it should verify the retrieved quote right before issuing the start(...).
Regarding the additional features or benefits you mentioned, I don't see that from this preliminary design, please advice, thanks.

@mssun As we discussed, It is quite challenging to maintain the security model for this integration because the GSGX is not designed to be integrated in API level, it has to be running in external processes, the secrets and encrypted workload/inputs/arguments are required to be exchanged with a new pre-load module which, in turns, needs to do the following tasks:

mutual attesting with Teaclave executor

establish a trusted channel with Teaclave executor

receive and parse the workload binaries and dependencies, input files, arguments, those are all complex objects.

deploy received workload binaries and dependencies, input files, arguments to GSGX enclave at runtime in correct layout and aligned with pre-defined GSGX manifest

the output of GSGX needs to be serialized and send back to Teaclave executor.

The pre-load module cannot manage the lifecycle of workload that is running in GSGX due to the limitation of GSGX, it has to be handled by Teaclave executor in external Linux signal mechanism, but it is not reliable and GSGX doesn't honor some signal type and still unstable to handle basic signals in GSGX

In addition, The GSGX is limited to run unique workload, it means it doesn't natively support measurement decoupling feature (refused by GSGX maintainer ITL), it implies a huge performance penalties and significant SGX resource consumption for FaaS usage scenario.


⚠ ClamAV has degraded performance

In 97866a5, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 223 ms

Resolved: ClamAV performance has improved in 9de890a.


Mock a command that's ran with command?

How would you mock a command that's ran with command?
Given a function that's under test
function foo
  command rm bar
  command ls
end

And you want to mock ls for example.
Doing mock ls 0 "echo foo" doesn't work, and mocking command is not possible because it's blacklisted.

Indeed is a good question, due to some limitations i am unable to mock some functions or commands, being command one of those.
Also I cannot mock the "runtime" created functions, so if you place a function inside your function in a nested scope, being that nested function the last one, it disables all the others.
I have to add some small features, like the possibility to use a custom function instead of a evaluated string, but other features might be a good call to arms if you want to help. Some help would be really nice.
If you have any idea or want to fiddle around and get in a point further where i did get i would be really grateful.

I see now why mock has a blacklist, because fish has reserved these names.
In fish 2.5.0 (but not 2.6.0 due to a regression), when you try to override a builtin
function: The name 'command' is reserved,
and can not be used as a function name

I'm not really sure how we can get around this limitation.
There used to be a --shadow-builtin option for functions but it was controversial and short-lived.


Change wording around legacy scope types

We've had users get confused into thinking that languages aren't properly supported because they only have legacy implementations.  I've reworded the docs so that it doesn't seem like legacy languages are not fully supported.  We can revert once we have more languages on next-gen scope engine
Checklist

[ ] I have added tests
[ ] I have updated the docs and cheatsheet
[ ] I have not broken the cheatsheet

@AndreasArvidsson hopefully we won't need this much longer due to https://github.com/cursorless-dev/cursorless/pull/2451 and the migrations, but figured I'd get this PR merged in in the meantime. Ready for review


it  stuck at ...cannot done

envy@ub1404:~/os_pri/github/NTM-tensorflow$ python main.py --task copy --is_train True
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally
{'checkpoint_dir': 'checkpoint',
'controller_layer_size': 1,
'epoch': 100000,
'input_dim': 10,
'is_train': True,
'max_length': 10,
'min_length': 1,
'output_dim': 10,
'read_head_size': 1,
'task': 'copy',
'test_max_length': 120,
'write_head_size': 1}
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:900] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 950M
major: 5 minor: 0 memoryClockRate (GHz) 1.124
pciBusID 0000:01:00.0
Total memory: 4.00GiB
Free memory: 3.56GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:755] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 950M, pci bus id: 0000:01:00.0)
[] Building a NTM model
Percent: [####################] 100.00% Finished.
[] Building a loss model for seq_length 1
[] Building a loss model for seq_length 2
[] Building a loss model for seq_length 3
[] Building a loss model for seq_length 4
[] Building a loss model for seq_length 5
[] Building a loss model for seq_length 6
[] Building a loss model for seq_length 7
[] Building a loss model for seq_length 8
[] Building a loss model for seq_length 9
[] Building a loss model for seq_length 10
[] Build a NTM model finished
[] Initialize all variables
[] Initialization finished
[    0]  4: 53.60 (618.3s)
[    5]  1: 13.75 (711.4s)
[   10]  4: 52.69 (711.8s)
[   15]  6: 79.43 (742.8s)
[   20]  1: 14.03 (757.4s)
.....................................................
[ 4775]  8: 0.00 (28215.3s)
[ 4780]  4: 0.00 (28216.3s)
[ 4785]  2: 0.00 (28216.9s)
[ 4790]  9: 0.00 (28218.1s)
[ 4795]  5: 0.00 (28218.9s)

What do you mean by stuck? I think the training starts well and it will end when it reach max epoch where 100000 is a default value.

Executing the original code, in my case it stucks earlier giving this output
python main.py --task copy --is_train True
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
{'checkpoint_dir': 'checkpoint',
'controller_layer_size': 1,
'epoch': 100000,
'input_dim': 10,
'is_train': True,
'max_length': 10,
'min_length': 1,
'output_dim': 10,
'read_head_size': 1,
'task': 'copy',
'test_max_length': 120,
'write_head_size': 1}
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:03:00.0
Total memory: 12.00GiB
Free memory: 11.87GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x2814190
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.82GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y Y
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 1:   Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
[] Building a NTM model
Percent: [####################] 100.00% Finished.
[] Building a loss model for seq_length 1
Traceback (most recent call last):
File "main.py", line 68, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 50, in main
cell, ntm = create_ntm(FLAGS, sess)
File "main.py", line 35, in create_ntm
test_max_length=FLAGS.test_max_length, **ntm_args)
File "/home/lina/deep_learning_frameworks/NTM-tensorflow-master/ntm.py", line 82, in init
self.build_model(forward_only)
File "/home/lina/deep_learning_frameworks/NTM-tensorflow-master/ntm.py", line 138, in build_model
binary_cross_entropy_with_logits)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/seq2seq.py", line 969, in sequence_loss
softmax_loss_function=softmax_loss_function))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/seq2seq.py", line 933, in sequence_loss_by_example
crossent = softmax_loss_function(logit, target)
File "/home/lina/deep_learning_frameworks/NTM-tensorflow-master/ops.py", line 217, in binary_cross_entropy_with_logits
with ops.name_scope(name, "bce_loss", [logits, targets]) as name:
TypeError: name_scope() takes exactly 1 argument (3 given)

In my case it stucks earlier and gives an output like this
python main.py --task copy --is_train True
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so locally
{'checkpoint_dir': 'checkpoint',
'controller_layer_size': 1,
'epoch': 100000,
'input_dim': 10,
'is_train': True,
'max_length': 10,
'min_length': 1,
'output_dim': 10,
'read_head_size': 1,
'task': 'copy',
'test_max_length': 120,
'write_head_size': 1}
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:03:00.0
Total memory: 12.00GiB
Free memory: 11.87GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:572] creating context when one is currently active; existing: 0x2814190
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 1 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:01:00.0
Total memory: 12.00GiB
Free memory: 11.82GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 1
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y Y
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 1:   Y Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:03:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:01:00.0)
[] Building a NTM model
Percent: [####################] 100.00% Finished.
[] Building a loss model for seq_length 1
Traceback (most recent call last):
File "main.py", line 68, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 50, in main
cell, ntm = create_ntm(FLAGS, sess)
File "main.py", line 35, in create_ntm
test_max_length=FLAGS.test_max_length, **ntm_args)
File "/home/lnmed/deep_learning_frameworks/NTM-tensorflow-master/ntm.py", line 82, in init
self.build_model(forward_only)
File "/home/lnmed/deep_learning_frameworks/NTM-tensorflow-master/ntm.py", line 138, in build_model
binary_cross_entropy_with_logits)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/seq2seq.py", line 969, in sequence_loss
softmax_loss_function=softmax_loss_function))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/seq2seq.py", line 933, in sequence_loss_by_example
crossent = softmax_loss_function(logit, target)
File "/home/lnmed/deep_learning_frameworks/NTM-tensorflow-master/ops.py", line 217, in binary_cross_entropy_with_logits
with ops.name_scope(name, "bce_loss", [logits, targets]) as name:
TypeError: name_scope() takes exactly 1 argument (3 given)


Redirect users to the current repo

It seems to me that the current repo is https://github.com/mehcode/python-saml
Redirecting users there would be best.

Yep, thanks.


Where's the document of this library?

As I look around and find no specific documentation about how to use datashape. Thus I don't know how to invoke its APIs and understand the input and output.
Even https://datashape.readthedocs.io/en/latest/index.html just tells the definition and grammar.
Or should I read the source codes directly?

I'm also having this problem. I found DataShape through a link on a StackOverflow answer but the documentation describes what DataShape does, but does not explain how to actually use it. How do I actually declare a type with DataShapes in my Python project?


Test: update DeleteCommandTest and DeleteCommandParserTest

Closes #93
Closes #94
DeleteCommand is 97% covered. Not sure where the missing line is as theres no red indicator on any lines 😢

@gigituang managed to resolve the issue, (refer to the commits in PR #133), ignore the latest merge because i screwed up the merge thing, but most of the changes should be there. I'll make some code suggestions later.
Managed to get 100% code coverage for DeleteCommand too 😎


ckpt file not saving when training has finished.

Once the training has finished and it goes to save the ckpt file it will tend to use all system RAM and not save the file.
Windows 10 | 16 Core AMD | 32G RAM | 3090
`Training halted. Summoning checkpoint as last.ckpt
Training complete. max_steps or max_epochs reached, or we blew up.
Traceback (most recent call last):
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 379, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 604, in _save
zip_file.write_record(name, storage.data_ptr(), num_bytes)
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 380, in save
return
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 259, in exit
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\caffe2\serialize\inline_container.cc:319] . unexpected pos<PHONE_NUMBER> vs<PHONE_NUMBER>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 754, in
trainer.fit(model, data)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 770, in fit
self._call_and_handle_interrupt(
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 723, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 811, in _fit_impl
results = self._run(model, ckpt_path=self.ckpt_path)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1236, in _run
results = self._run_stage()
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1323, in _run_stage
return self._run_train()
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1353, in _run_train
self.fit_loop.run()
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\loops\base.py", line 205, in run
self.on_advance_end()
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\loops\fit_loop.py", line 294, in on_advance_end
self.trainer._call_callback_hooks("on_train_epoch_end")
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1636, in _call_callback_hooks
fn(self, self.lightning_module, *args, **kwargs)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 308, in on_train_epoch_end
self._save_topk_checkpoint(trainer, monitor_candidates)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 379, in _save_topk_checkpoint
self._save_monitor_checkpoint(trainer, monitor_candidates)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 651, in _save_monitor_checkpoint
self._update_best_and_save(current, trainer, monitor_candidates)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 702, in _update_best_and_save
self._save_checkpoint(trainer, filepath)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\callbacks\model_checkpoint.py", line 384, in _save_checkpoint
trainer.save_checkpoint(filepath, self.save_weights_only)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 2467, in save_checkpoint
self._checkpoint_connector.save_checkpoint(filepath, weights_only=weights_only, storage_options=storage_options)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 445, in save_checkpoint
self.trainer.strategy.save_checkpoint(_checkpoint, filepath, storage_options=storage_options)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 418, in save_checkpoint
self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\plugins\io\torch_plugin.py", line 54, in save_checkpoint
atomic_save(checkpoint, path)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\utilities\cloud_io.py", line 67, in atomic_save
torch.save(checkpoint, bytesbuffer)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 381, in save
_legacy_save(obj, opened_file, pickle_module, pickle_protocol)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 225, in exit
self.file_like.flush()
ValueError: I/O operation on closed file.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 379, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 604, in _save
zip_file.write_record(name, storage.data_ptr(), num_bytes)
MemoryError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 380, in save
return
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 259, in exit
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\caffe2\serialize\inline_container.cc:319] . unexpected pos<PHONE_NUMBER> vs<PHONE_NUMBER>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 756, in
melk()
File "main.py", line 733, in melk
trainer.save_checkpoint(ckpt_path)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 2467, in save_checkpoint
self._checkpoint_connector.save_checkpoint(filepath, weights_only=weights_only, storage_options=storage_options)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\trainer\connectors\checkpoint_connector.py", line 445, in save_checkpoint
self.trainer.strategy.save_checkpoint(_checkpoint, filepath, storage_options=storage_options)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\strategies\strategy.py", line 418, in save_checkpoint
self.checkpoint_io.save_checkpoint(checkpoint, filepath, storage_options=storage_options)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\plugins\io\torch_plugin.py", line 54, in save_checkpoint
atomic_save(checkpoint, path)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\pytorch_lightning\utilities\cloud_io.py", line 67, in atomic_save
torch.save(checkpoint, bytesbuffer)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 381, in save
_legacy_save(obj, opened_file, pickle_module, pickle_protocol)
File "C:\Users\User01\anaconda3\envs\everydream\lib\site-packages\torch\serialization.py", line 225, in exit
self.file_like.flush()
ValueError: I/O operation on closed file.
`

I'm getting the same thing, doesn't make it past the first epoch, but a last.cpkt did get saved for me...


[Inf Durability] Fused weapons will still break with infinite durability cheat.

During my playthrough of totk I had the infinite durability cheat turned on.  I confirmed it was working with a few tree branches in the tutorial.  However, I noticed that my fused weapon of two bokoblin arms still broke when hitting wooden boxes.  Not sure if this is an issue specific to two base weapons fused together, or if it still happens with a weapon + item fusion.  Thank you all for your work so far!

Wanted to confirm - this isn't just with wooden boxes/panels, it seems like weapons fused with weapons will eventually break, even with the cheat turned on

Please send save file and description of how to duplicate error/bug.

It also appears to occur when fusing a zonai device to a sword, though I'm unsure of the speed at which it damages the item.

Equip a brand new weapon, such as an Soldier's Broadsord (flash in top right indicating unused).
Drop a zonai device, such as a flame emitter.
Fuse the zonai device onto the weapon.
Swing the weapon a few times. Upon checking the menu, you will see that the flashing sparkle indicating a brand new weapon has disappeared.

This only appears to affect weapons; shields with zonai devices attached appear to remain brand new.
slot_01.zip

Can you test this..
[#01. test durability]
080E0000 02B1FFF8 D65F03C0 B9000901
040E0000 02B1FFF4 B9401901
040E0000 006920EC 948444A4
040E0000 006920EC 9484456E```

@F-Lambda / @Alceris
[#06. Inf. Equipment Durability (Fire will still break it.)]
040E0000 01B832FC 1B09A3E8
080E0000 02B1FFB4 D65F03C0 B9000901
040E0000 02B1FFB0 B9401901
040E0000 00A0ED64 94844493
040E0000 00A0EA3C 9484455D


Fix DTO nested field renaming

Pull Request Checklist

[ ] New code has 100% test coverage
[ ] (If applicable) The prose documentation has been updated to reflect the changes introduced by this PR
[ ] (If applicable) The reference documentation has been updated to reflect the changes introduced by this PR

By submitting this issue, you agree to:

follow Litestar's Code of Conduct
follow Litestar's Contribution Guidelines

Description
Currently nested fields cannot be renamed or excluded. This PR implements this behaviour.
Close Issue(s)

Hey @abdulhaq-e - I'd already made a start on the nested excludes, so I pushed that through in #1596.
Solution you are using here looks good to me for as long as we don't support nested rename_fields. Is there something you are waiting on before converting from draft?

Nope, all good, just wanted to check the approach first.


Proposal: multitask and transfer learning

Create a benchmark for multi task learning (beginning with performance across Atari games), then develop a reinforcement learning architecture that beats existing techniques on the benchmark. The benchmark will be able to measure both absolute performance on all tasks (e.g. top scores on all Atari games) and also how much more quickly a given architecture learns an unseen task when it is first trained on unrelated tasks (e.g. ratio of score trained vs. untrained on a new game and a fixed number of frames).
Rendered view of the proposal

Well formatted, I love this topic, I'm currently going after a similar research project for lab-scientific environments with limited supervision : )!
In regards to the question that the project is going after, this might be either too broad or too narrow.
In terms of creating only a benchmark, using already existing benchmarks to start from would be wiser, then moving on to try new hybrid architectures on the environments, since after all this is directed towards transfer learning & multi-task learning.
For example, currently, the environments provided for (world_of_bits) soon tend to have benchmark released for transfer learning, ( as mentioned here ). Using something like this as a starting point could be more productive.
Here is another relevant work from Berkley RL lab, Benchmarking Deep Reinforcement Learning for Continuous Control, which contains various benchmarks that can be built on top.
Now in terms of the transfer learning, I looked at the given related work, those are good, however to answer the transfer learning approach, there have been more solid ground work in that area to look at first (which could be directly applied for questions proposed as research):

Reinforcement Learning with Unsupervised Auxiliary Tasks, introduces the (UNREAL) agent to combine main agent and auxiliary tasks learned (which could be a shared task for several environments)

Learning to Navigate in Complex Environments, introduces (The online multi learner algorithm for multi task learning), which is a neural network-based agents that have modular structures and that are trained on multiple tasks, with inputs coming from different modalities (vision, depth, past rewards and past
actions).

Meanwhile on a similar track, but for real world environments, here is another paper mentioning a semi-supervised approach to RL agents, in favor of generalizing skills that used / happens in most environments Generalizing Skills with Semi-Supervised Reinforcement Learning. On Page 2, Figure, there is a comparison how:

while in transfer learning, the policy is finetuned using a known reward function in the unlabeled MDP set. Semi-supervised RL is distinct in that it involves using experience from the unlabeled set without access to the reward function.

and then, the authors point out:

Note, however, that the method that we develop could be used for transfer learning problems where the state and reward are consistent
across domains.

More on the robotics and real world transfer learning, this paper suggests few other ways to learn a maze, other than having access to navigational information and path, Deep Reinforcement Learning with Successor Features for Navigation across Similar Environments, similar approaches can be introduced to specific (limited) environments.

bonus: Transfer learning can be introduced through different fashions, not just as actor critic method style. The notion of having multi-agents exploring new areas as for transfer learning seem to be a more feasible approach to be expanded on. In this regards,  Learning to Communicate with Deep Multi-Agent Reinforcement Learning, introduces how agents and communicate to improves rewards functions when one of them might taking a leap.

The topic is interesting, lots of active research based results are coming out, but for this proposal, I would suggest leaning more towards one way more, since it can be too broad to go after replicating / doing all those tasks in one research.
The applied research version of this would be creating an RL-Agent that reuses one of those mentioned state-of-art benchmark papers and improve up-on it.

Thanks for taking the time to make such a thorough response, I really appreciate it!
Ok, so re:other benchmarks. The biggest issue I have with existing benchmarks, for this purpose, is that they measure absolute reward on one game, not how much knowledge is transferred between games. Individual papers will measure it, but they do it in effectively ad-hoc ways that are hard to compare across papers. The kind of benchmark I want to make is intrinsically about doing cross-validation after training on different subsets of environments. That's not something I've seen done yet, but I think it's the best approach to measuring transfer, since RL environments tend to be scarce. So the benchmark is more about having a nice baseline for how to measure progress across games with wildly varying amounts of absolute reward. Ultimately, I don't think it'll take too long to build, since the hard parts would be coding the environments themselves and I'm definitely reusing those.
Also, as a side-note I think it's methodologically important to specify up front what the measuring stick is, before coming up with architectures.
As for architectures themselves, I have a bunch of ideas. I think the recent paper Loss is its own reward and predictron both point the way towards solving the sparse reward issues in RL by coming up with auxiliary rewards that incidentally build representations that are reuseable. But I don't want to flesh the ideas out too much until the benchmark is in place for the aforementioned reasons.

Sounds good! Since it's in the Fundamental research already, there can be room to wiggle the problem around.
Maybe update the information to include parts of the last message you posted, as in:

Individual papers will measure it, but they do it in effectively ad-hoc ways that are hard to compare across papers. The kind of benchmark I want to make is intrinsically about doing cross-validation after training on different subsets of environments.

and

Also, as a side-note I think it's methodologically important to specify up front what the measuring stick is, before coming up with architectures.

The Predictron is pretty good, I'm reviewing that paper this weekend!
I'm guessing as a starting point:

Determining the environments
Benchmarks to look for
and lastly including the results / or related papers to the chosen environments would be cool.

I'm happy to help and find the relevant work in that regard and try getting the benchmarks.

Ok cool, I've updated the proposal a bit, hopefully made it clearer about the goals, and I've added the UNREAL paper to the bottom though I haven't had time to add the summary. I'll be digging into all of the papers you suggested and I'll add them as I go

LGTM, merging. Thanks @0bserver07 for reviewing!

Absolutely :)!

Please take note of the formatting changes I did: https://github.com/AI-ON/ai-on.org/commit/dc3fb4b48af5127b30409e5467fd6414284ddca3


Registering caching together with session

Adding session without adding caching always throws on first temp data access:
public void Configure(IApplicationBuilder app)
{
    app.UseSession();
    // Other code
}
public void ConfigureServices(IServiceCollection services)
{
    services.AddSession();
    // Other code
}

maybe AddSession could register caching by default if it's required anyways?
Because dependency injection says very little about what is wrong with me codez :octocat:

It doesn't add caching because it doesn't know what distributed cache you want to use. E.g. memory, redis, sql, etc.

Couldn't it add memory by default with ability to override?

Should I close this as won't fix?

@Muchiachio I don't know if using memory cache by default is a good idea. Closing as won't fix.


UI: fix templates/offerings are not listed in autoscale vm profile when login as user

Description
This PR fixes an UI issue that regular users cannot list offering and templates for autoscale vm group
Steps to reproduce the issue

log in as regular user
create autoscale vm group
go to the vm group details, click "AutoScale VM Profile" tab.

click 'Edit AutoScale VM Profile"

With this PR

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[x] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale

[ ] Major
[ ] Minor

Bug Severity

[ ] BLOCKER
[ ] Critical
[x] Major
[ ] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

The root cause of the issue:
(localcloud) 🐱 > list templates templatefilter=all listall=true filter=id,name
🙈 Error: (HTTP 431, error code 4350) Filter all can be specified by admin only

(localcloud) 🐱 > list serviceofferings listall=true isrecursive=true issystem=false filter=id,name
🙈 Error: (HTTP 431, error code 4350) Only ROOT admins and Domain admins can list service offerings with isrecursive=true

@blueorangutan package

Packaging result: :heavy_check_mark: el7 :heavy_check_mark: el8 :heavy_check_mark: el9 :heavy_check_mark: debian :heavy_check_mark: suse15. SL-JID 5725

@blueorangutan ui

@harikrishna-patnala a Jenkins job has been kicked to build UI QA env. I'll keep you posted as I make progress.

UI build: :heavy_check_mark:
Live QA URL: https://qa.cloudstack.cloud/simulator/pr/7343 (QA-JID-60)

@NuxRo
I think you have tested this change, right ? could you approve it ? Thanks

CLGTM


mwc-formfield allowing hyperlinks inside of label

Is your feature request related to a problem? Please describe.
Using mwc-formfield with mwc-checkbox in "I accept Terms of Service and Privacy Policy" scenarios
Describe the solution you'd like
Using either named slot or attribute to set label for mwc-checkbox
Describe alternatives you've considered
none
Additional context
example of checkbox label with links inside of it: https://hypothes.is/signup

  <input type="checkbox" class="form-checkbox__input" name="privacy_accepted" id="deformField5" value="true" required="">

      I have read and agree to the privacy policy, terms of service, and community guidelines.

I just wanted to ask if that feature was already considered by project team. Thanks!

This feature actually has been considered already and asked for internally (Googler context: b/111314020 yaqs/6488321546256384).
The main reason we are not supporting <slot> for labels in our components is because the burden of maintenance is increased on us to support more arbitrary use cases. For example, we would need to support hyperlink styling, touch targets, accessibility, and more just with this one UX.
Instead, our approach is to release baseline Material components that are extremely strict to spec, while allowing users to extend components and add additional custom or Material-inspired functionality on top of the baseline.
See https://github.com/material-components/material-components-web-components/issues/602 for more details. That issue will be the one to track for this feature. I'll update it to mention labels.


Add 'retryWhen' operator

Hey guys, I want to add feature that retryWhen operator with retry operator's conditional promise  to determine whether retry is possible even if not reached maximum retry count.
Promise<Int> { (resolve, reject, _) in
    currentAttempt += 1
    if currentAttempt < successOnAttempt {
        print("attempt is \(currentAttempt)... reject")
        reject(TestErrors.anotherError)
    } else {
        print("attempt is \(currentAttempt)... resolve")
        resolve(5)
    }
}.retryWhen(retryAttempts) { (remainAttempts, error) -> Promise<Bool> in
    if remainAttempts > retryableRemainAttempt {
        print("retry remainAttempts is \(remainAttempts)... true")
        return Promise<Bool>(resolved: true).defer(5) // I want here promise chain.
    } else {
        print("retry remainAttempts is \(remainAttempts)... false")
        return Promise<Bool>(resolved: false)
    }
}

Nice addition, thanks!


Can't add host if the host running external scheduler went down and was not re-scheduled to another host due to no healthchecks being active in Rancher

Rancher Versions:
1.4
If the host which run external scheduler went down, the scheduler is still reported as healthy and active in cattle database, Thus when adding another host, all the microservice containers on the second host will still find the dead scheduler to delegate scheduler requests. That will cause those container stuck at scheduling phase and failed to start because the dead scheduler can't reply to those events.

@chrisurwin has verified this is fixed in v1.4.1-rc1


Move liveview outside Javis

Moving Liveview to a Different Package JavisViewer

I think this issue was tagged by mistake in #468 @gpucce


docs: response override

Fixes #4658
Release Notes: No

thanks @zhaohuabing, can you also add this doc to docs/ and v1.2/


ValidatablePickerControl

Hey,
I am trying to extend the functionality to include a validatablePickerControl. I have this largely working except i cannot pass through the ItemDisplayBinding from the xaml. Below is some code snippets. Any pointers greatly appreciated.
    public static BindableProperty ItemDisplayBindingProperty = BindableProperty.Create(
        nameof(ItemDisplayBinding),
        typeof(BindingBase),
        typeof(ValidatablePicker));

    public BindingBase ItemDisplayBinding
    {
        get => _picker.ItemDisplayBinding;
        set => _picker.ItemDisplayBinding = value;
    }
    /// <summary>
    /// </summary>
    public ValidatablePicker()
    {
        _picker.BindingContext = this;
        _picker.SetBinding(Picker.ItemsSourceProperty, nameof(ItemsSource));
        _picker.SetBinding(Picker.TitleProperty, nameof(Title));
        _picker.SetBinding(Picker.SelectedIndexProperty, nameof(SelectedIndex));

        _picker.ItemDisplayBinding = ItemDisplayBinding;

        Children.Add(_picker);

        _errorLabel.SetBinding(IsVisibleProperty, nameof(IsInValid));
        _errorLabel.SetBinding(Label.TextProperty, nameof(Message));
        _errorLabel.BindingContext = this;
        _errorLabel.TextColor = Color.Red;

        Children.Add(_errorLabel);
    }

                    <controls:ValidatablePicker
                SelectedIndex="{Binding Model.TypeSelectedIndex, Mode=TwoWay}"
                ItemsSource="{Binding AssetTypes}"
                ItemDisplayBinding="{Binding Description}"
                BindingName="AssetTypeId"
                Title="Select asset type"
                />

Hey @dooie,
Hard to tell without the full sample of ValidatabalePicker, but at first glance I think that the ItemDisplayBinding should look something like:
public BindingBase ItemDisplay         {             get             {                 return (BindingBase)GetValue(ItemDisplayProperty);             }             set             {                 SetValue(ItemDisplayProperty, value);             }         }
Then in the constructor you'd use the "SetBinding" method from "_picker" as the other three declarations.
If that doesn't work, I'll try to recreate the scenario locally in the next few day(s).

Hey @dooie,
Hard to tell without the full sample of ValidatabalePicker, but at first glance I think that the ItemDisplayBinding should look something like:
public BindingBase ItemDisplay { get { return (BindingBase)GetValue(ItemDisplayProperty); } set { SetValue(ItemDisplayProperty, value); } }
Then in the constructor you'd use the "SetBinding" method from "_picker" as the other three declarations.
If that doesn't work, I'll try to recreate the scenario locally in the next few day(s).

Yeah, 'SetBinding' only works on the static bindable properties of the picker. ItemDisplayBinding is simply a public BindingBase property?

@dooie For my own knowledge, I'm going to dig into why they decided to not make that a bindable property.  My guess is that is has something to do with the order of operations for Android vs Apple.
Nevertheless, I managed to find a solution that doesn't involve extending the current Picker or creating a custom renderer:
1.) Add an optional parameter to your control as a string
public ValidatablePicker(string itemDisplay = "") { ... }
2.) Set the property to a new Binding with the value from constructor.
_picker.ItemDisplayBinding = new Binding(itemDisplay);
3.) In Xaml, pass the value you want using x:Arguments like so:
<custom:ValidatablePicker ItemsSource="{Binding Source}"> <x:Arguments>                     <x:String>Name</x:String> <--- "Description" would go here                 </x:Arguments>  </custom:ValidatablePicker>

@dooie Another solution I found thanks to this Xamarin Forum would be the following:

Change

public static BindableProperty ItemDisplayBindingProperty = BindableProperty.Create(
        nameof(ItemDisplayBinding),
        typeof(BindingBase),
        typeof(ValidatablePicker));

to
public static BindableProperty ItemDisplayBindingProperty = BindableProperty.CreateAttached(
        "ItemDisplayBinding",
        typeof(string),
        typeof(ValidatablePicker),
        default(string),
        BindingMode.OneWay,
        propertyChanged: (bindable, oldValue, newValue) =>
        {
            var vp = (ValidatablePicker)bindable;
            vp._picker.ItemDisplayBinding = new Binding(newValue.ToString());
        });

Notice that I've changed the Create to CreateAttached.  Also, I've added the setting of the ItemDisplayBinding in the propertyChanged event that occurs.

In the XAML you would then just set the property to whatever string you want, for example:

<custom:ValidatablePicker ItemsSource="{Binding Source}" ItemDisplay="Description" />

Remove the following:

    public BindingBase ItemDisplayBinding
    {
        get => _picker.ItemDisplayBinding;
        set => _picker.ItemDisplayBinding = value;
    }

@dooie Another solution I found thanks to this Xamarin Forum would be the following:

Change

public static BindableProperty ItemDisplayBindingProperty = BindableProperty.Create(
        nameof(ItemDisplayBinding),
        typeof(BindingBase),
        typeof(ValidatablePicker));

to
public static BindableProperty ItemDisplayBindingProperty = BindableProperty.CreateAttached(
        "ItemDisplayBinding",
        typeof(string),
        typeof(ValidatablePicker),
        default(string),
        BindingMode.OneWay,
        propertyChanged: (bindable, oldValue, newValue) =>
        {
            var vp = (ValidatablePicker)bindable;
            vp._picker.ItemDisplayBinding = new Binding(newValue.ToString());
        });

Notice that I've changed the Create to CreateAttached. Also, I've added the setting of the ItemDisplayBinding in the propertyChanged event that occurs.

In the XAML you would then just set the property to whatever string you want, for example:

<custom:ValidatablePicker ItemsSource="{Binding Source}" ItemDisplay="Description" />

Remove the following:

    public BindingBase ItemDisplayBinding
    {
        get => _picker.ItemDisplayBinding;
        set => _picker.ItemDisplayBinding = value;
    }

@james-russo Works a treat, thank you!

@dooie my pleasure!  If you feel like it, feel free to create a PR for the control and we'll get reviewed and merged in for others! :)

@dooie my pleasure! If you feel like it, feel free to create a PR for the control and we'll get reviewed and merged in for others! :)

Ill look to do a pull over the weekend

Full exemple since I think it take me two hour to figure this out once I've read this article.
<?xml version="1.0" encoding="UTF-8"?>
<ContentView xmlns="http://schemas.microsoft.com/dotnet/2021/maui"
             xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"
             x:Class="PGEN.Maui.Controls.CustomEntry"
             xmlns:local="clr-namespace:PGEN.Maui.Converters"
             x:Name="this">
    <ContentView.Resources>
        <local:ErrorMessageToUnderlineColorConverter x:Key="errorMessageToUnderlineColorConverter" />
    </ContentView.Resources>
    <StackLayout BindingContext="{x:Reference this}"
            Orientation="Vertical"
            HorizontalOptions="Center"
            VerticalOptions="Center"
            WidthRequest="{Binding WidthRequest, FallbackValue='300'}">

        <Entry Style="{StaticResource EntryForm}"
               Text="{Binding Text}"
               HorizontalOptions="FillAndExpand" />

        <Line BackgroundColor="{Binding IsNotValidMessage, Converter={StaticResource errorMessageToUnderlineColorConverter}}"
              HeightRequest="2"
              HorizontalOptions="FillAndExpand"
              />
        <Label TextColor="Red"
               FontAttributes="Bold"
               Text="{Binding IsNotValidMessage}"
               MinimumHeightRequest="20"
               HeightRequest="25"
               HorizontalOptions="FillAndExpand"  />
    </StackLayout>
</ContentView>

namespace PGEN.Maui.Controls
{
    public partial class CustomEntry : ContentView
    {
        public static readonly BindableProperty TextProperty = BindableProperty.Create(
            nameof(Text),
            typeof(string),
            typeof(CustomEntry),
            string.Empty,
            defaultBindingMode: BindingMode.TwoWay,
            propertyChanging: OnTextChange);

        public static readonly BindableProperty IsNotValidMessageProperty = BindableProperty.Create(
            nameof(IsNotValidMessage),
            typeof(string),
            typeof(CustomEntry),
            null,
            propertyChanging: OnIsNotValidMessageChange,
            propertyChanged: OnIsNotValidMessageChanged);

        public string Text
        {
            get => (string)GetValue(TextProperty);
            set => SetValue(TextProperty, value);
        }

        public string IsNotValidMessage
        {
            get => (string)GetValue(IsNotValidMessageProperty);
            set => SetValue(IsNotValidMessageProperty, value);
        }

        public CustomEntry()
        {
            InitializeComponent();
        }

        static void OnTextChange(BindableObject bindable, object oldValue, object newValue)
        {
            //var s = "";
           //((TextBox)bindable).IsValid = (bool)newValue;
        }

        static void OnIsNotValidMessageChange(BindableObject bindable, object oldValue, object newValue)
        {
            var s = "";
            //((TextBox)bindable).IsNotValidMessage = (string)newValue;
        }

        static void OnIsNotValidMessageChanged(BindableObject bindable, object oldValue, object newValue)
        {
            var s = "";
            //((TextBox)bindable).IsNotValidMessage = (string)newValue;
        }
    }
}


Is there any guide for adding providers or source for novels?

I was wondering if there was any template or example for adding a provider or source. I would find it easy to contribute if there were some steps given in the readme for adding another provider.

ye that would be nice tbh
The basic idea of this app is to scrape a website by constructing the get requests ourself and using css selectors (and json) to then extract the data then returning a standard response containing all the data the app needs like author, chapters, ect
Here is the relevant info for scraping
https://github.com/Blatzar/scraping-tutorial
Here is the class you must override https://github.com/LagradOst/QuickNovel/blob/master/app/src/main/java/com/lagradost/quicknovel/MainAPI.kt#L14
You only need to override search (for searching), load (to show the data from a searchresult url) and loadHtml (to load a chapter and return the html of the text) to make a real provider
both loadMainPage (lets user sort a list of the most popular novels defined by mainCategories, orderBys and tags) and loadReviews (adds reviews to the load response) are optional
to test your provider just add it to the list https://github.com/LagradOst/QuickNovel/blob/aad6c927f8999de9dab73f743ff58c263a976ff4/app/src/main/java/com/lagradost/quicknovel/util/Apis.kt#L19 and run the app (you need android studio to compile it)

If you have any questions or need any help then just ask. If you feel like the api is lacking or you want to add some feature (like recommendations of other novels on the result page) then just create an issue and I will have the api updated


Do not parse new TLS 1.3 extensions outside of tests

All new extensions should be gated, just to avoid unexpected
issues in production before we've finished and tested 1.3.
This change was motivated by finding a problem where the new
supported versions extension can break RSA key exchange when
the client supports TLS1.3 but negotiates down to TLS1.2 and
tries to use RSA key exchange.
Issue # (if available):
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

:exclamation: No coverage uploaded for pull request head (gate_extensions@e5de362). Click here to learn what that means.
The diff coverage is n/a.


add 16 new tlds

these are epic sigma tlds (accept pls)

https://github.com/face-hh/webx/pull/66#issuecomment-2143349184

sad :(


BigQuery: Can't create a table with a expirationtime

Hello,
I am trying to create a table using the bigquery API in the following way:
final Long expirationTime = getExpirationTimeMillis(dateString);
      final TableInfo tableInfo = TableInfo.newBuilder(TableId.of(project, dataset, tableId), NEW_TABLE_DEF)
          .setExpirationTime(expirationTime)
          .build();
      LOG.info(String.format("Creating table: %s.%s.%s", project, dataset, tableId));
      return bigQuery.create(tableInfo);

This gives me the following message:
com.google.cloud.bigquery.BigQueryException: Not found: Table entity-dev:joel_daily.user_daily_20170718
Doing the same without .setExpirationTime(expirationTime) works fine. I don't see why I should get an issue Not found when setting the ExpirationTime.
Can you please help me with this issue?

@joelostlund can you check to make sure the expiration time you're passing is not in the past?

@tcoffee-google yeah it worked, thanks. Must say that this is a little bit confusing this you specify the days from now when you create a dataset and now+the days from now when creating a table.

@joelostlund I agree that in the context of an "expiration time" that is a bit confusing. I'll pass this along to the API team.


[4.19] Linking not occurring while Editor is open

From my testing it seems linking / hotloading does not take place while the UE4 editor is open. Compiles ok, but does not link. By closing the editor and compiling again, linking is performed like normal.
Does not happen always, but quite often.
[Windows 10 / VS 2015 / Haxe 3.4.7]

How are you compiling it? By pressing the compile button on the editor or
by compiling through Visual Studio?
On Mon, Apr 23, 2018, 06:37 Datee<EMAIL_ADDRESS>wrote:

From my testing it seems linking / hotloading does not take place while
the UE4 editor is open. Compiles ok, but does not link. By closing the
editor and compiling again, linking is performed like normal.
Does not happen always, but quite often.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/proletariatgames/unreal.hx/issues/85, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAK3-22Gj4PGEE0CupBt1KqG8i1mZfocks5traDtgaJpZM4Tfjwh
.

I'm building from commandline, but the issue is the same with VS. Pressing the compile button in the Editor causes a hotload to happen, but the code changes are not picked up. Closing the editor and compiling again and the linking is performed as it should..

As far as I know, if you build on visual studio/command-line, it will try to replace your current dll, which will indeed fail in the linking phase. I don't think this has anything to do with Unreal.hx though, and would be the same in a project without it. Could you confirm that?
There is a command-line argument you can pass on so that it compiles it with a suffix. I'm not sure what's the argument, but you can see it when clicking on the compile button on the editor.
What is more mysterious is why your changes are not getting picked up when clicking on the compile button inside the editor. My theory is that it is related to the previous failure - compiling with the compile button adds a -SkipLink option which may skip linking if no changes are detected. The issue is that since unreal.hx compiles the haxe code to a static library, it is not detected as a change and UBT skips linking in this case. We work around this issue by touching a small .cpp file every time there is a change to the static library. My theory is that once the first compilation fails at linking phase (because you are compiling inside visual studio), the next builds don't detect any change (because of the SkipLink option).
You can see if that's the case by  deleting your Intermediate/Haxe directory, and then compiling again - this time through the editor's compile button.

Thanks for the tests! Could you give me an example of what you are changing, as well as the build logs?
Thanks!

I'm on 4.21 now and the issue is still the same.. any updates to this? :)

Hi! Sorry for the long wait before answering this! What does tools\tasks\BuildTask.bat BuildUE4Project release SideScroller do?

Its just a bat script to compile from commandline really :
"%ENGINE_PATH%\Engine\Build\BatchFiles\Build.bat" %PROJECT_NAME%%EDITOR_BUILD_CONFIG% %EDITOR_BUILD_PLATFORM% %EDITOR_BUILD_TARGET% %PROJECT_FILE% -waitmutex -%VISUAL_STUDIO_VERSION%


⚠ Bofrost.at has degraded performance

In 3832b6d, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1737 ms

Resolved: Bofrost.at performance has improved in fbe4813 after 21 minutes.


Improve the FontLoader code

Please refer to the individual commit messages.

/botio test

Thank you for simplifying this!

Thanks for landing this, and other patches!

With respect to the first commit, to me it's quite fascinating just how easy it is to accidentally leave unused code around when making changes and how slow finding those cases can be.
And in general, given how much code I've been removing and re-factoring over the years, it still amazes me how code that's not even touched seem to somehow "age".


2261 endpoints for storage and messaging config setup status

Closes #2261
https://github.com/ethyca/fidesdocs/issues/56 has been created to track doc updates related to this and other related functionality updates.
Code Changes

GET /messaging/default/status endpoint that returns a high-level "status" of messaging configuration

messaging is considered successfully configured iff the notifications.notification_service_type property is set and it points to a messaging service type with a (default) configuration that has all necessary details and secrets populated

similar GET /storage/default/status endpoint that returns a high-level "status" of storage configuration

storage is considered successfully configured iff the storage.active_default_storage_type property (only settable through API) is set to s3 and a default s3 storage configuration is present and fully populated with necessary details and secrets (if applicable)

response to both endpoints looks like the following:

{
  "config_status": "configured",
  "detail": "Active default [storage location/messaging service] of type [s3/MAILGUN] is fully configured"
}

or
{
  "config_status": "not_configured",
  "detail": "No active default [messaging/storage] configuration found"
}

Steps to Confirm

run your fides server
try hitting the endpoint(s) in various different states, with messaging/storage fully configured or not
some useful endpoints to update storage and messaging config states in order to evaluate the endpoints added here:

PATCH /api/v1/config
GET /api/v1/config?api_set=true
PUT /api/v1/storage/default
PUT /api/v1/storage/default/{storage_type}/secret
GET /api/v1/storage/default/active
PUT /api/v1/messaging/default
PUT /api/v1/messaging/default/{service_type}/secret
GET /api/v1/messaging/default/active

Pre-Merge Checklist

[ ] All CI Pipelines Succeeded
Documentation:

[ ] documentation complete, PR opened in fidesdocs
[x] documentation issue created in fidesdocs

[x] Issue Requirements are Met
[ ] Relevant Follow-Up Issues Created
[x] Update CHANGELOG.md

Description Of Changes
AC in the issue describes what qualifies as "fully configured" for both messaging and storage -- copied here:

Complete for storage configuration is true only if S3 configuration (with secrets) is complete AND is set as the active default
Complete for messaging configuration is true if the active default messaging type has all of its configuration details (secrets and necessary connection details) configured

thanks for the quick review @eastandwestwind! i'll look to address your code comments, i think they're worth addressing and the timing here can afford a few iterations.
for docs, i've created this ticket https://github.com/ethyca/fidesdocs/issues/56 and will be addressing all the storage/messaging updates wholesale as part of that.
i'll look to get a postman update in place, but if it's slipping too much then i'll add in a note to cover these endpoints to the follow-up we'd created for the other messaging/storage work!

@eastandwestwind got code updates up addressing your comments, let me know your thoughts on them when you've got a chance.
in the meantime i'll look to update the postman collection!


Fix ridge regression example: change x0 to init in solve_cg

The current ridge regression example is broken with the following error:
TypeError: jax._src.scipy.sparse.linalg.cg() got multiple values for keyword argument 'x0'

I believe this stems from the fact that linear_solve.solve_cg takes init as opposed to x0 as a named parameter
https://github.com/google/jaxopt/blob/e42a1ff6f80ca2a3dae5fa60c2a7fedad91d527c/jaxopt/_src/linear_solve.py#L109

@googlebot I signed it!

Good catch! Thanks a lot.


Statically link cpu backend into ngraph shared library

Hi Robert,
Cool but just to check : loading both the cpu backend (to be soon embedded inside ngraph) and the cuda/intelgpu/cldnn backend (via dynload) via the same instance could potentially load some unfriendly libs : example : iomp vs gomp.
Have you just checked that the cuda/intelgpu/cldnn backends are compatible with the cpu backend, meaning both could be loaded at the same time without potential conflicts/interference ?
Cheers
Ref:
ldd /tmp/ngraph/lib/libcpu_backend.so
linux-vdso.so.1 =>  (0x00007ffe28d87000)
libngraph.so.0.5 => /tmp/ngraph/lib/libngraph.so.0.5 (0x00007fbcdce79000)
libcodegen.so.0.5 => /tmp/ngraph/lib/libcodegen.so.0.5 (0x00007fbcd8a6e000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbcd8851000)
libmkldnn.so.0 => /tmp/ngraph/lib/libmkldnn.so.0 (0x00007fbcd80fa000)
libmklml_intel.so => /tmp/ngraph/lib/libmklml_intel.so (0x00007fbccf029000)
libtbb.so.2 => /tmp/ngraph/lib/libtbb.so.2 (0x00007fbccedeb000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fbccea69000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbcce760000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fbcce54a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbcce180000)
/lib64/ld-linux-x86-64.so.2 (0x00007fbcdfbfc000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbccdf7c000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fbccdd53000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fbccdb39000)
libiomp5.so => /tmp/ngraph/lib/libiomp5.so (0x00007fbccd795000)

@WilliamTambellini Thanks for the heads up. We do currently run CI tests with CPU and Nvidia GPU enabled in the same build and we are getting a test environment setup for the Intel GPU CI.
This PR is to get things working with TensorFlow integration.


Query annotations for queries with joins that return multiple rows.

Version
1.12.0
What happened?
When I join two tables and try to select everything from only one of them using the generated code return only one item rather than a list.
Relevant log output
No response
Database schema
-- Example queries for sqlc
create table church_events
(
    id           bigserial
        primary key,
    created_at   timestamp with time zone,
    updated_at   timestamp with time zone,
    deleted_at   timestamp with time zone,
    organizer_id bigint                   not null
        constraint fk_church_events_organizer
            references users
            on update cascade on delete set null,
    cover_image  text                     not null,
    title        text                     not null,
    sub_title    text                     not null,
    content      text                     not null,
    scheduled_on timestamp with time zone not null,
    age          integer                  not null
);

create table church_jobs
(
    duty            text   not null,
    church_event_id bigint not null
        constraint fk_church_event
            references church_events
            on update cascade on delete set null,
    id              bigserial
        primary key,
    created_at      timestamp with time zone not null,,
    updated_at      timestamp with time zone,
    deleted_at      timestamp with time zone
);

SQL queries
-- name: GetEvent :one
select *
from church_events
    left join   church_jobs cj on church_events.id = cj.church_event_id
where church_events.deleted_at IS NULL
  AND church_events.id = $1;

Configuration
{
  "version": "1",
  "packages": [
    {
      "path": "db",
      "engine": "postgresql",
      "schema": "query.sql",
      "queries": "query.sql"
    }
  ]
}

Playground URL
https://play.sqlc.dev/p/751428fd61df6380a97a365d86e61818671ea15494a0df5cbf0d5b3ff7de48f0
What operating system are you using?
Linux
What database engines are you using?
PostgreSQL
What type of code are you generating?
Go

hello sir the issue is closed but the bug is still there.suppose i want to return an struct with the following structure
` type ChurchEvent struct {
ID          int64
CreatedAt   sql.NullTime
UpdatedAt   sql.NullTime
DeletedAt   sql.NullTime
OrganizerID int64
CoverImage  string
Title       string
SubTitle    string
Content     string
ScheduledOn time.Time
Age         int32
[] ChurchJob struct {
Duty          string
ChurchEventID int64
ID            int64
CreatedAt     sql.NullTime
UpdatedAt     sql.NullTime
DeletedAt     sql.NullTime
}
}
`
what would you suppose i do..?

Change
-- name: GetEvent :one
select *
...

to
 -- name: GetEvent :many
select *
...

adding :many  seems to change nothing ..i want to return a struct  with an embedded slice of type struct.

oh, as far as I know, that is not possible.
I'm usually using s repository that uses sqlc and it reshapes the structs the way I need them.
Your DB will fetch the rows and sqlc maps tgem to the result 1:1. afaik it doesn't do any reshaping. you're not defining custom structs, sqlc generates them.

This issue should not be closed!

I agree, this should not be closed.
I'm writing a Select query using embed that fetches an object and it's many sub-objects (i.e. 1:many relation) via a join, and the resulting row struct has only one object and one sub-object, when it should have one object and many sub-objects.
Expectation:
type ExampleRow struct {
	Object Object
	SubObject []SubObject
}

Actual:
type ExampleRow struct {
	Object Object
	SubObject SubObject
}


Idea

Make the option so it saves the changes u made even if u go in different menu's, u can disable it and enable it in the settings if added.

thats not what devtools is for, you can create a happy textures tp to do this though

I dont know how thats why


Cleaned up SonarCloud code smells as well as IntelliJ warnings in the…

… fhirengine directory and test directory
This PR clean up code smells found by SonarCloud and warnings found by IntelloiJ in the fhirengine directory and test directory.
If you are suggesting a fix for a currently exploitable issue, please disclose the issue to the prime-reportstream team directly outside of GitHub instead of filing a PR, so we may immediately patch the affected systems before a disclosure. See SECURITY.md/Reporting a Vulnerability for more information.
Test Steps:

Look at the issues in the SonarCloud report and make sure that any code smells for the fhirengine directory that are remaining have tickets written up for them. (See ticket here https://app.zenhub.com/workspaces/platform-6182b02547c1130010f459db/issues/gh/cdcgov/prime-reportstream/10049)
Make sure the unit tests still pass.
Open the project in IntelliJ and make sure all warnings are gone (there are a couple for http vs https that cannot be changed however because that is the url of the system ex loinc's website)

Changes
Cleans up code smells found by SonarCloud and warnings found by IntelloiJ in the fhirengine directory and test directory.
Checklist
Testing

[ ] Tested locally?
[ ] Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
[ ] (For Changes to /frontend-react/...) Ran npm run lint:write?
[ ] Added tests?

Process

[ ] Are there licensing issues with any new dependencies introduced?
[ ] Includes a summary of what a code reviewer should test/verify?
[ ] Updated the release notes?
[ ] Database changes are submitted as a separate PR?
[ ] DevOps team has been notified if PR requires ops support?

Linked Issues

Fixes #issue

To Be Done
Create GitHub issues to track the work remaining, if any

#issue

Specific Security-related subjects a reviewer should pay specific attention to

Does this PR introduce new endpoints?

new endpoint A
new endpoint B

Does this PR include changes in authentication and/or authorization of existing endpoints?
Does this change introduce new dependencies that need vetting?
Does this change require changes to our infrastructure?
Does logging contain sensitive data?
Does this PR include or remove any sensitive information itself?

If you answered 'yes' to any of the questions above, conduct a detailed Review that addresses at least:

What are the potential security threats and mitigations? Please list the STRIDE threats and how they are mitigated

Spoofing (faking authenticity)

Threat T, which could be achieved by A, is mitigated by M

Tampering (influence or sabotage the integrity of information, data, or system)
Repudiation (the ability to dispute the origin or originator of an action)
Information disclosure (data made available to entities who should not have it)
Denial of service (make a resource unavailable)
Elevation of Privilege (reduce restrictions that apply or gain privileges one should not have)

Have you ensured logging does not contain sensitive data?
Have you received any additional approvals needed for this change?

Changes look good to me, I think you got all the sonar cloud smells from an eye balling of the links on the ticket.
I did notice this warning Actual value of parameter 'taskAction' is always 'TaskAction.process' in FhirConverterTests

Good catch, updated and ready for your review @arnejduranovic


Support for compute capability 8.6

Hi, I have tried to compile torchvision for rtx3070 and cuda 11.0, but without any luck. The code I am using is:
USE_CUDA=1 USE_CUDNN=1 USE_MKLDNN=1 TORCH_CUDA_ARCH_LIST="8.6" python setup.py install
But I am getting error that this compute capability is not supported. How can I make it compile? Thanks

Compute capability 8.6 is supported in CUDA>=11.1, so you would need to update your local CUDA toolkit or compile with 8.0+PTX.

@ptrblck does it mean that we don't need to do anything in particular in torchvision? Would you say we can close this issue?

@fmassa I don't think you need to change anything in torchvision to build with 8.6, as we are doing it in our containers with CUDA11.1 and 8.6 without any changes.
I think we can close this issue.

ok, thanks for the clarification

@batrlatom as @ptrblck told me, to compile with 8.0+PTX, you can set TORCH_CUDA_ARCH_LIST="8.0+PTX" with your current cuda 11.0

I was able to compile pytorch witch cuda 11.1 and everything works as it should. Thanks!


CLOUDP-196436 Check podman installation and machine init/start

Proposed changes
Check podman installation and run podman machine init and podman machine start

Jira ticket: CLOUDP-196436

Checklist

[ ] I have signed the MongoDB CLA
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added any necessary documentation in document requirements section listed in CONTRIBUTING.md (if appropriate)
[ ] I have addressed the @mongodb/docs-cloud-team comments (if appropriate)
[ ] I have updated test/README.md (if an e2e test has been added)
[ ] I have run make fmt and formatted my code

Further comments

evergreen status


Bug Report

Game name: (please, specify the game name)
Engine version: (specify engine version, if known)
Platform: (specify platform, if known)
Error message: Some data left
Stack trace:    в uTinyRipper.Lz4.Lz4DecodeStream.ReadBuffer(Stream stream, Int64 count)
в uTinyRipper.BundleFileScheme.ReadFileStreamMetadata(Stream stream, Int64 basePosition)
в uTinyRipper.BundleFileScheme.ReadScheme(Stream stream)
в uTinyRipper.GameCollection.ReadScheme(SmartStream stream, String filePath, String fileName)
в uTinyRipper.GameCollection.LoadScheme(String filePath, String fileName)
в uTinyRipper.GameStructure.ProcessPlatformStructure(GameStructureProcessor processor, PlatformGameStructure structure)
в uTinyRipper.GameStructure.Load(List1 pathes, LayoutInfo layinfo)    в uTinyRipper.GameStructure.Load(IEnumerable1 pathes, LayoutInfo layinfo)
в uTinyRipperGUI.MainWindow.LoadFiles(Object data)

Your game uses the Addressable system which is not supported by uTinyRipper.

But same of assets is can be unlocked bu uTinyRipper
Чт, 22 июля 2021 г. в 11:03, Jeremy Pritts @.***>:

Your game uses the Addressable system which is not supported by
uTinyRipper.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mafaca/UtinyRipper/issues/958#issuecomment-884727015,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AECBAFKQXEKQ6CWLEKTUXHTTY7GGLANCNFSM5ASIVMAA
.

Not supported

Support for Unity versions up to 2020.2.0 has been recently added to my AssetRipper fork.


Restore measuretype.valid_until_used field

E.g., for sandbags.

This means that also the housemeasure table needs a used_in_round field.

To make the fields in the housemeasure table more clear, change round_number to bought_in_round.

ALTER TABLE `housinggame`.`housemeasure`
ADD COLUMN `used_in_round` INT(11) NOT NULL DEFAULT -1 AFTER `housegroup_id`,
CHANGE COLUMN `round_number` `bought_in_round` INT(10) UNSIGNED NOT NULL ;

ALTER TABLE `housinggame`.`measuretype`
ADD COLUMN `valid_until_used` TINYINT(3) NOT NULL DEFAULT 0 AFTER `valid_one_round`;


Roll Flutter Engine from 0a14c519ea4f to 54757dab9462 (1 revision)

https://github.com/flutter/engine/compare/0a14c519ea4f...54757dab9462
2024-09-11<EMAIL_ADDRESS>Roll Dart SDK from 072945e0b457 to a438066d634f (1 revision) (flutter/engine#55100)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC aaclarke@google.com,rmistry<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


Worked On the Registration page and dashboard screen

Please verify and merged the code in master

Verify and Merged


expressed, chorizontist

This pull request was generated by the 'mq' tool
[test]
flake rate: 0.4
logical conflict every: 100
sleep for: 300s
[pullrequest]
requests per hour: 30
deps=[c,e]

/trunk merge


[prisma-test-utils] PostgreSQL: WrongType

Some databases tested via prisma-test-utils (in Introspection CI) have a type not yet supported by Prisma.
Errors looks like this (example standup-slack-bot):
2020-03-11T13:58:27.129Z prisma-client Prisma Client call:
2020-03-11T13:58:27.130Z prisma-client prisma.eventids.create({
  data: {
    created_at: new Date('2100-05-30T04:43:46.598Z'),
    event_id: '4c2cba28-f4c6-55aa-9680-dac0c4e43a1a'
  }
})
2020-03-11T13:58:27.130Z prisma-client Generated request:
2020-03-11T13:58:27.130Z prisma-client mutation {
  createOneeventids(data: {
    created_at: "2100-05-30T04:43:46.598Z"
    event_id: "4c2cba28-f4c6-55aa-9680-dac0c4e43a1a"
  }) {
    created_at
    event_id
  }
}

...

Error occurred during query execution:
ConnectorError(ConnectorError { user_facing_error: None, kind: QueryError(Error { kind: FromSql(0), cause: Some(WrongType { postgres: Type(Timestamptz), rust: "core::option::Option<alloc::string::String>" }) }) })
    at PrismaClientFetcher.request (/home/runner/work/introspection-engine-output/introspection-engine-output/clients/postgresql_public_test_utils/standup-slack-bot/runtime/index.js:1:47203)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

model eventids {
  created_at DateTime
  event_id   String   @id
}

CREATE TABLE public.eventids (
    event_id text NOT NULL,
    created_at timestamp with time zone DEFAULT now() NOT NULL
);

Internal notes:
Specific type is listed in brackets next to the database log link.
Test utils database(s) affected:

https://github.com/prisma/introspection-engine-output/blob/test_utils/test-utils/results/postgresql_public_test_utils_niko-public-1_stderr.log (timestampz)
https://github.com/prisma/introspection-engine-output/blob/test_utils/test-utils/results/postgresql_public_test_utils_sonarqube_stderr.log  (bytea)
https://github.com/prisma/introspection-engine-output/blob/test_utils/test-utils/results/postgresql_public_test_utils_standup-slack-bot_stderr.log (timestampz)
https://github.com/prisma/introspection-engine-output/blob/test_utils/test-utils/results/postgresql_public_test_utils_odoo_stderr.log (bytea)

I did some manual testing - as of alpha 927 this is fixed, after the postgres types mapping work.

Can you confirm in test-utils @divyenduz?

As of 928. I can't see the timestampz error anymore but I can see see the  bytea error. As it stands:

Internal notes:
Specific type is listed in brackets next to the database log link.
Test utils database(s) affected:

https://github.com/prisma/introspection-engine-output/blob/test_utils_928/test-utils/results/postgresql_public_test_utils_sonarqube_stderr.log

This is not reproducible anymore with 2.1.0-dev.49. Sonarqube now fails with https://github.com/prisma/prisma/issues/3007 instead.


ngx_pagespeed shows that is working properly , but in browser i see no results of it doing so

This is my current configuration
pagespeed on;
pagespeed FileCachePath /var/ngx_pagespeed_cache;
pagespeed RewriteLevel CoreFilters;
pagespeed EnableFilters collapse_whitespace;
pagespeed EnableFilters combine_css;
pagespeed EnableFilters combine_javascript;
Serf sends requests too css and js files [01/Feb/2017:15:59:30 +0000] "GET /lib/ionic/css/ionicons.css HTTP/1.1" 200 60946 "http://localhost/" "Serf/1.1.0 mod_pagespeed/<IP_ADDRESS>-4238"
And even tho it says it works properly  , only the collapse_whitespace actually collapses white space...
The others seem like they just don't do anything .

It may take a few page loads before you get fully optimized responses.
Also, If you add ?PageSpeedFilters=+debug on the querystring, the module will add debug notes in html comments about what prevents it from optimizing images/css/js files, if any.
Furthermore, there is the admin interface, which contains logging and a console which may be helpful in diagnosing problems. https://modpagespeed.com/doc/admin
Lastly, there is the f.a.q. which may be helpful in getting started: https://modpagespeed.com/doc/faq

If we have one service on ec2 instance does that mean that we need to install pagespeed on every instance/service?

I am not sure what you are asking, could you elaborate? Do you have multiple nginx servers on multiple machines servicing requests to the same hostname?

yes we have multiple AWS EC2 server machines with nginx on every machine and nodejs service application in a docker container. And  we have ionic web app  in a docker container, again, on other machine where we have installed nginx pagespeed. If we request the url which servicing other machine it says: Uncaught ReferenceError: pagespeed is not defined.

I am not sure I fully understand the setup, but this sounds a bit like https://github.com/pagespeed/ngx_pagespeed/issues/1219
Could you try disabling combine_javascript, to see if the eliminates the problem?
pagespeed DisableFilters combine_javascript;

combine_javascript is disabled and this what happens in debug mode:

So it seems that the module is in passthrough mode.
In the config you posted you had:
pagespeed EnableFilters collapse_whitespace;
pagespeed EnableFilters combine_css;

Did you remove that from the configuration? I would have expected these filters to be enabled, and an indication of them being enable in the debug comment you posted.
Also, I would be very surprised if you still get  Uncaught ReferenceError: pagespeed is not defined. Are you still seeing that error?

We have the same Uncaught ReferenceError: pagespeed is not defined
even with config combine_javascript  disabled
Our whole config:
# === PageSpeed === #
#
# https://modpagespeed.com/doc/configuration

pagespeed on;

# Ensure requests for pagespeed optimized resources go to the pagespeed handler
# and no extraneous headers get set.
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
  add_header "" "";
}
location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }

# Ensure relative URLs not to be rewritten to absolute URLs
pagespeed PreserveUrlRelativity on;

# Better performance with lowercase keywords in HTML
pagespeed LowercaseHtmlNames on;

# Remove extra whitespaces in HTML documents
# Safe to be turned on as long as CSS property `white-space` unused
pagespeed EnableFilters collapse_whitespace;

# Change
# <button name="ok" disabled="disabled">
# to
# <button name="ok" disabled>
pagespeed EnableFilters elide_attributes;

# Add resource preloading HTTP headers
#
# Spec: http://w3c.github.io/preload/
# Browser Support: http://caniuse.com/#search=preload
#
# https://github.com/pagespeed/ngx_pagespeed/issues/1408
pagespeed EnableFilters hint_preload_subresources;

# Add DNS prefetch tags
#
# Sometimes not working
# https://github.com/pagespeed/ngx_pagespeed/issues/541
pagespeed EnableFilters insert_dns_prefetch;

# Remove unnecessary quotes
pagespeed EnableFilters remove_quotes;

# Trim absolute URLs to relative URLs if possible
# Disabled since might have some risks
# (not quite worth it compared to benefit)
#
# pagespeed EnableFilters trim_urls;

# Change style tag with @import only
# To link tags
# Useful for blog maybe, main website don't have any though
#
pagespeed EnableFilters inline_import_to_link;

# Disable rewriting (mostly minification) CSS
# since we already does minification
# and this filter will change caching period to `max-age=300,private`
#
pagespeed DisableFilters rewrite_css;

# Disable rewriting (mostly minification) JS
# since we already does minification
# and this filter will change caching period to `max-age=300,private`
#
pagespeed DisableFilters rewrite_javascript_external;

# Disable combining JS
# since we already does that in assets pipeline
# and this filter will change caching period to `max-age=300,private`
pagespeed DisableFilters combine_javascript;

# Minifying inline JS is fine since no caching involved
pagespeed EnableFilters rewrite_javascript_inline;

# From original description in
# https://modpagespeed.com/doc/filter-css-rewrite
#
# The CSS parser cannot parse some CSS3 or proprietary CSS extensions.
# If fallback_rewrite_css_urls is not enabled,
# these CSS files will not be rewritten at all.
# If the fallback_rewrite_css_urls filter is enabled,
# a fallback method will attempt to rewrite the URLs in the CSS file,
# even if the CSS cannot be successfully parsed and minified.
#
pagespeed ForbidFilters fallback_rewrite_css_urls;

# Enable rewriting style attribute for image urls
#
pagespeed EnableFilters rewrite_style_attributes_with_url;

# Flatten @import in CSS files
#
pagespeed EnableFilters flatten_css_imports;

# Inline content of small CSS files into HTML
#
pagespeed EnableFilters inline_css;

# Inline content of small JS files into HTML
#
pagespeed EnableFilters inline_javascript;

# Move CSS tags above script tags
# Faster rendering
#
pagespeed EnableFilters move_css_above_scripts;

# Remove HTML comments
# Except Internet Explorer conditional comments
#
pagespeed EnableFilters remove_comments;

# 'Extend Cache' seeks to improve the cacheability of a web page's resources
# without compromising the ability of site owners to change the resources and
# have those changes propagate to users' browsers.
#
pagespeed EnableFilters extend_cache_images;
pagespeed EnableFilters extend_cache_scripts;
pagespeed EnableFilters extend_cache_css;

# Convert images into formats with better performance
#
pagespeed EnableFilters convert_jpeg_to_progressive;
pagespeed EnableFilters convert_jpeg_to_webp;
pagespeed EnableFilters convert_png_to_jpeg;
pagespeed EnableFilters convert_to_webp_animated;
pagespeed EnableFilters convert_to_webp_lossless;
pagespeed EnableFilters inline_images;
pagespeed EnableFilters inline_preview_images;
pagespeed EnableFilters resize_mobile_images;
pagespeed EnableFilters recompress_jpeg;
pagespeed EnableFilters recompress_png;
pagespeed EnableFilters recompress_webp;
pagespeed EnableFilters jpeg_subsampling;
pagespeed EnableFilters resize_images;
pagespeed EnableFilters strip_image_color_profile;
pagespeed EnableFilters strip_image_meta_data;

# Set s-maxage for unoptimized resources
# 86400s = 24 hours = 1 day
pagespeed InPlaceSMaxAgeSec 86400;

# Disable CSS combine since it rewrites protocol relative URLs to HTTP only
pagespeed ForbidFilters combine_css;

# === PageSpeed === #


Unconnected WireGuard clients shown in pivpn -c

The command 'pivpn -c' returns the list of connected clients as expected. However, the clients are still show when user calls 'pivpn -c' after the clients have already disconnected. Is this a limitation of WireGuard or am I missing something?
Expected behavior: 'pivpn -c' returns only currently connected clients, not disconnected ones.

I think this behavior is OK for me, because it's show when the client was seen connected last time where you can see the time. Maybe, but I have only two clients, but if there is many of them, maybe sort them by this last seen time, but like I said for me it's OK because you can see when they was connected with server last time.
After help command they write show connected clients, this can be little bit misleading, but I like to see the time last connection and statistics like it works now.

That's how WireGuard works. There is not a concept of a "connection", more a "handshake on demand". A client will perform the hanshake with the server when it has to send data. It you don't send data you can't tell if the client is off or it has nothing to send. You will see those clients in pivpn -c until you reboot or restart the wireguard interface.

Thanks for the clarification. I was used to the way OpenVPN handled things hence my hesitation. Closing the issue now.


Create login page

Project
Create login page
Issue Type:
Feature Request
Comments
Not sure if new page features are what you're looking for but thought I'd suggest it anyways!

Ok! You can start working on it! @rebekahpidgeon


object has no attribute '_container' in DockerContainer.del(self)

Describe the bug
AFTER a pipeline run in azure dev ops (which had a failure for the dockerpy + requests/urllib3 incompatibility), I get the following errors:
======================= 14 passed, 131 errors in 24.94s ========================
Exception ignored in: <function DockerContainer.__del__ at 0x7f0d9990da20>
Traceback (most recent call last):
  File "/home/vsts/.cache/pypoetry/virtualenvs/mds-data-core-3AFF4dIC-py3.10/lib/python3.10/site-packages/testcontainers/core/container.py", line 86, in __del__
    if self._container is not None:
AttributeError: 'PostgresContainer' object has no attribute '_container'
Exception ignored in: <function DockerContainer.__del__ at 0x7f0d9990da20>
Traceback (most recent call last):
  File "/home/vsts/.cache/pypoetry/virtualenvs/mds-data-core-3AFF4dIC-py3.10/lib/python3.10/site-packages/testcontainers/core/container.py", line 86, in __del__
    if self._container is not None:
AttributeError: 'ConfluentKafkaContainer' object has no attribute '_container'

ConfluentKafkaContainer is my workaorund for #351
To me this looks like some parts of the class are already gone when the __del__ is called?
The following packages were installed during the run:
testcontainers-kafka = {version = "^0.0.1rc1"}
testcontainers-postgres = {version = "^0.0.1rc1"}
testcontainers-compose = {version = "^0.0.1rc1"}

The python docs on del() have this:

del() can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to None. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the del() method is called.


No reference to how to use configs

File: engine/reference/commandline/config.md, CC @mstanleyjones
Google brought me here but looking at these docs, there is no link to https://docs.docker.com/engine/swarm/configs/ for how to actually use them.

Closed via #4072