🐛 BUG: spinner prevents git GPG code signing

Which Cloudflare product(s) does this pertain to?
C3
What version(s) of the tool(s) are you using?
2.11.0
What version of Node are you using?
v20.11.0
What operating system and version are you using?
Linux
Describe the Bug
Observed behavior
npm create<EMAIL_ADDRESS>-- hello-world --type hello-world --ts --git
using create-cloudflare version 2.11.0

╭ Create an application with Cloudflare Step 1 of 3
│
├ In which directory do you want to create your application?
│ dir ./hello-world
│
├ What type of application do you want to create?
│ type "Hello World" Worker
│
├ Do you want to use TypeScript?
│ yes typescript
│
├ Copying template files
│ files copied to project directory
│
├ Updating `package.json`
│ updated `package.json`
│
├ Installing dependencies
│ installed via `npm install`
│
╰ Application created

╭ Configuring your application for Cloudflare Step 2 of 3
│
├ Installing @cloudflare/workers-types
│ installed via npm
│
├ Adding latest types to `tsconfig.json`
│ added @cloudflare/workers-types/2023-07-01
│
├ Retrieving current workerd compatibility date
│ compatibility date 2024-01-29
│
├ Do you want to use git for version control?
│ yes git
│
Please enter the passphrase to unlock the OpenPGP secret key:
...

└ Committing new files ..

The Committing new files above is constantly smashing the terminal with an animation, thus preventing clear input of the GPG password.
Expected behavior
Turn off the spinner when interacting with the user.
If the spinner can't detect when the user is being prompted for a GPG password, then at least add a --no-useSpinner argument to the CLI.
Steps to reproduce
See above.
Please provide a link to a minimal reproduction
No response
Please provide any relevant error logs
No response

@jculvey do you need anything else to fix this?
The spinner negatively affects both pinentry-tty and pinentry-curses. Each in a different bad way.
Workaround is to unlock the gpg signing key before running c3 with its silly spinner.
signingkey=$(git config get user.signingkey)
if [[ "$signingkey" ]]
then gpg --sign --local-user $signingkey </dev/null >/dev/null
fi

Better solution would be to add a --no-useSpinner option.
Best would be to detect that gpg/gpagent is prompting the user for input, and get out of the way.


Clarify required vs optional checkbox questions and 0 boxes checked

Describe the bug
For checkbox questions, admins might think that applicants can check 0 boxes on a required checkbox question and that that's a valid response. However, it looks like it is not:

In the Comprehensive Sample Program, I tried answering all questions except the "Which of the following kitchen instruments do you own?" checkbox question and then clicking Save&next, but I got the "This question is required" error.
I also tried changing that question on the admin side to allow 0 as the minimum number of choices required, but I still got the "This question is required" error as an applicant.

We should make it clearer to admins that if a checkbox question is marked as required, then at least one option must be chosen. We can do this by:

Mentioning under checkbox question here that if you want to allow no options to be selected, the question should be optional
Don't allow 0 as the minimum on the admin UI and instead say it should be set as optional if we expect people to not select anything.

See https://github.com/civiform/civiform/pull/6992#discussion_r1541201844.

(Removing bug cause there's not an actual software issue, right?)

Updated the CiviForm docs, but leaving this open in case we also want to change wording on the admin UI.


Fullscreenview share button

When a user selects the slideshow and the slideshow becomes fullscreen. would like to add in an export button. I have written some code to do that however when when the button is pressed the fullscreenview is dismissed and the activityviewcontroller is never presented.
Share.frame = CGRect(x: self.view.center.x - 25, y: 25, width: 50, height: 50)
        Share.setImage(#imageLiteral(resourceName: "upload"), for: UIControlState())
        Share.addTarget(self, action: #selector(FullScreenSlideshowViewController.close), for: UIControlEvents.touchUpInside)

        view.addSubview(Share)

@objc func actionshare() {
        let img: UIImage = slideshow.currentSlideshowItem?.image as! UIImage
        let shareItems:Array = [img]
        let activityViewController:UIActivityViewController = UIActivityViewController(activityItems: shareItems, applicationActivities: nil)
        activityViewController.excludedActivityTypes = [UIActivityType.print, UIActivityType.postToWeibo, UIActivityType.copyToPasteboard, UIActivityType.addToReadingList, UIActivityType.postToVimeo]
        self.present(activityViewController, animated: true, completion: nil)
    }

I just want to note, that it could crash with Could not cast value of type 'xxx.NukeSource4ImageSlideshow' to 'UIImage'


On Windows, /dev/null causes E_UNKNOWN error.

https://github.com/3scale/3scale_toolbox/blob/dbdbda74a18ecf7af5026cebd48338786198ba35/lib/3scale_toolbox/commands/product_command/import_command.rb#L31
  "code": "E_UNKNOWN",
  "message": "No such file or directory @ rb_sysopen - /dev/null",
  "class": "Errno::ENOENT",

Hi @leathersole,
Thanks for reporting this.
I've opened a PR with a fix for it in https://github.com/3scale/3scale_toolbox/pull/290


List command's linespec option does not work in other frames.

What version of Delve are you using (dlv version)?

[vagrant@k8s-env apply]$ dlv version
Delve Debugger
Version: 0.11.0-alpha
Build:

What version of Go are you using? (go version)?

[vagrant@k8s-env apply]$ go version
go version go1.7.4 linux/amd64

What operating system and processor architecture are you using?

[vagrant@k8s-env apply]$ uname -a
Linux k8s-env 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

What did you do?

Using list command in a different frame with the linespec parameter.

What did you expect to see?

A new set of lines printed

What did you see instead?

Same set of lines get printed as if I did not pass linespec
For a more detailed example. While debugging kubernetes/pkg/kubectl/cmd/apply_test.go
I ended up in a segfault with 38 deep stack.
The list command in a different frame did not take the linespec parameter into consideration
(dlv) frame 7 ls
   431:			return 0, EOF
   432:		}
   433:		if int64(len(p)) > l.N {
   434:			p = p[0:l.N]
   435:		}
=> 436:		n, err = l.R.Read(p)
   437:		l.N -= int64(n)
   438:		return
   439:	}
   440:
   441:	// NewSectionReader returns a SectionReader that reads from r
(dlv) frame 7 ls 441
   431:			return 0, EOF
   432:		}
   433:		if int64(len(p)) > l.N {
   434:			p = p[0:l.N]
   435:		}
=> 436:		n, err = l.R.Read(p)
   437:		l.N -= int64(n)
   438:		return
   439:	}
   440:
   441:	// NewSectionReader returns a SectionReader that reads from r

On the current frame the linespec parameter does take effect
(dlv) ls
> [unrecovered-panic] runtime.startpanic() /usr/local/go/src/runtime/panic.go:543 (hits goroutine(14):1 total:1) (PC: 0x42ccc0)
   538:		}
   539:		return nil
   540:	}
   541:
   542:	//go:nosplit
=> 543:	func startpanic() {
   544:		systemstack(startpanic_m)
   545:	}
   546:
   547:	//go:nosplit
   548:	func dopanic(unused int) {
(dlv) ls 548
 543:	func startpanic() {
 544:		systemstack(startpanic_m)
 545:	}
 546:
 547:	//go:nosplit
 548:	func dopanic(unused int) {
 549:		pc := getcallerpc(unsafe.Pointer(&unused))
 550:		sp := getcallersp(unsafe.Pointer(&unused))
 551:		gp := getg()
 552:		systemstack(func() {
 553:			dopanic_m(gp, pc, sp) // should never return
(dlv)

Fixed by https://github.com/derekparker/delve/pull/796#pullrequestreview-33560643


Fix Very Big Value Bug for Expired Tokens.

Issue #, if available:
None.
Description of changes:
If the DynamoDB result contains a LastEvaluatedKey, then the ExpiredTokens result array will contain an additional array. This is a workaround as ExpiredTokens cannot be counted correctly in that case.

Hi @kght6123
The issue causing an additional array to appear in ExpiredTokens endpoint result has been identified and resolved. The fix will be included in the upcoming patch release. Once the release containing the fix is out, this PR will be closed.

Hi @kght6123
The fix for the issue has been included in release 1.1.3. As a result, we are closing this PR. Please feel free to verify the resolution, and don't hesitate to reach out if you encounter any further issues.
Thank you for your contribution!


Kafka S3 connection + refactor of separate Snowflake + S3 connector deployment

The github actions profile has some issue


[11.x] Add Request::fluent method

Description
This PR adds a new fluent() method to the Illuminate\Http\Request class.
Similar to the Request::collect method, being able to transform the request input to a Fluent instance helps in being able to transport the input data around your application (such as an action or a queued job) and provides you null-safe access instead of using an array.
Usage
Consider an action that consumes a Fluent instance to access request data safely that may be nullable, while also being portable for easy testing without having to provide a request instance:
use App\Actions\CreateComment;

class CommentController extends Controller
{
    public function store(CommentRequest $request)
    {
        $comment = (new CreateComment)->handle(
            $request->fluent()
        );

        return response($comment);
    }
}

class StoreComment extends QueuedJob
{
    public function __construct(
        public Fluent $data
    ) {}

    public function handle(): Comment
    {
        return Comment::create([
            'title' => $this->data->title,
            'body' => $this->data->body,
            // ...
        ]);
    }
}

Nice 👌 what're your thoughts on a FluentRequest class that can be used to type hint the controller function param to allow auto-injection of the fluent wrapped version, avoiding need for the manual call to ->fluent()?
Easy enough to write in userland I guess, but a nicety I'd probably never get round to in my projects.

@GavG I'm not sure if that'd be easily do-able, and/or worth the pay-off, imo. How would it work?

@GavG I'm not sure if that'd be easily do-able, and/or worth the pay-off, imo. How would it work?

Having had a look through, I'd agree, it's not worth the pay off.
It could be done when resolving controller method params, but it wouldn't be very clean (casting to a subclass 👎).
Resolving it earlier on also wouldn't be feasible with the way Laravel routing currently works.
So I retract my idea, for the sake of saving one line of code 😂

@GavG haha okay sounds good! Appreciate the discussion though.


W0D5 updates

W0D5 Statistics Tutorials post-production editing

Pending reviewer and final little edits linked here (6/1/21).


[GraphQL API] 'amplify status api -acm' CLI command not supported for flutter?

Description
While debugging a separate issue, I attempted to use the Amplify CLI control matrix report as documented here: https://docs.amplify.aws/cli/graphql/authorization-rules/#advanced
The report does not print out anything about the model control matrix; output is just as follows:

I attempted to run this from the repo root and from the amplify API directory. Am I running this command wrong, or is this feature not supported for Flutter?
Also, is there a list of features supported by Amplify CLI that aren't supported in Amplify Flutter? I've been logging issues, but wondering if there's a quicker way for me to check and make sure the features are supposed to be supported before logging an issue?
Categories

[ ] Analytics
[ ] API (REST)
[X] API (GraphQL)
[ ] Auth
[ ] Authenticator
[ ] DataStore
[ ] Storage

Steps to Reproduce

Go to a project repository with API enabled
Run 'amplify status api -acm '
See that the output is not as expected.

Screenshots
No response
Platforms

[X] iOS
[X] Android

Android Device/Emulator API Level
No response
Environment
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.0.0, on macOS 12.3.1 21E258 darwin-x64, locale en-US)
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
[✓] Xcode - develop for iOS and macOS (Xcode 13.3.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 4.2)
[✓] IntelliJ IDEA Community Edition (version 2020.3.2)
[✓] VS Code (version 1.67.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

• No issues found!

Dependencies
N/A

Device
N/A
OS
N/A
CLI Version
5.3.1
Additional Context
Excluding dependencies, because it's a long list and unnecessary/unrelated to the problem since the problem is an Amplify CLI issue.

Hey @lfraker :wave: thanks for raising this! As @HuiSF pointed out this may be due to an older version of the CLI where this was either not working as expected or not supported at the time. What version of the CLI are you using?

@HuiSF thanks for pointing that out!
@josefaidt definitely looks like I was using an older version of the CLI. I had run 'amplify upgrade' but hadn't upgraded via npm. Upgrading now and seeing if that resolves it! Will update shortly.

@josefaidt that was the issue; closing this issue. Sorry for the confusion!


Blue giga issue 77

Bluegiga implementation for use with BLED112-dongle

Hi @dkurok ... Awesome work. Your Pull Request is now so much more ... C# stylish ;)

Regards master branch / git: As long as the pull request is still building and has no conflicts, I can squash it without issues.
You re-code some of the CLI/Example toolings to use now BlueGiga... that I cannot merge currently (since default on Win10 is WinRT for most devices and I do not yet have dynamic loading on Linux. How about introducing an argument to select the bluetooth adapter

After that, give me a sign and I merge it.

Sorry, something did go wrong with my branching!
It generated a new PR on your side; I thought I first merged my own branch into my main and then I could do a PR / it updates the PR by itself...but NO, it generated a new PR on your site (https://github.com/sharpbrick/powered-up/pull/145)
So I think you can delete this (#140 ) and only need to treat on 145

Happy git learning. Git and GitHub are really different animal. When you want to experiment, create local branches on your cloned repo. No Pull Request. A "Pull Request" is when you say your code is ready into the main repo and request that I pull it. Basically you offer a patch.

Closed in favor of #145


when Deploying the contract on remix getting error

What do I do wrong in this token code?
pragma solidity ^0.4.16;
interface tokenRecipient { function receiveApproval(address _from, uint256 _value, address _token, bytes _extraData) public; }
Error:
Functions in interfaces should be declared external.
interface tokenRecipient { function receiveApproval(address _from, uint256 _value, address _token, bytes _extraData) public; }

interface tokenRecipient { function receiveApproval(address _from, uint256 _value, address _token, bytes _extraData) external; } would remove the warning


PHOENIX-3547 Supporting more number of indices per table.

Currently the number of indices per Phoenix table is bound to maximum of 65,535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices.
This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtedly big enough to cover this requirement.
Any existing Phoenix table will still continue to support only maximum of 65535 of indices.
A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's viewIndex data type.
On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to Long while this value would be Short for any existing table.
According to Protobuf documentation https://developers.google.com/protocol-buffers/docs/proto#updating we can change the type of viewIndexId from int32 to int64 and maintain the backward compatibility for older clients. We did a manual verification for this scenario by
creating a view with the old client and verify the new client is able to connect and read/write to the view after changing the viewIndexId type and adding the viewIndexType.

@twdsilva I can see the Jenkins job is complaining about the some lines being  bigger than 100 chars. Many of those had been there and my commit just changed part of it. I can fix them all but not sure if this is expected from this PR.
Also I see some tests that had been reported to fail but they pass on retries.
Can you you give me an update on the status and what are things expected here.
Thanks

@twdsilva all done 👍

@m2je  I was able to apply this PR to the 4.x branches, but it didn't apply cleanly to the master branch. Can you please attach a patch the the JIRA that applies cleanly to the master branch ?

@twdsilva Sure, The only issue I'm seeing is the each 4.x branch also need a separated patches. Let me prepare them.

@twdsilva Added all patches to the ticket. Unfortunately they seem to all need to have their own patch except  4.x-cdh5.12, 4.x-cdh5.13,4.x-cdh5.14. But I added a patch for each anyway.
I have also captured the work here in case you want me to open pull requests for review purposes or comparing changes.

https://github.com/apache/phoenix/compare/4.x-HBase-0.98...m2je:4.x-HBase-0.98-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-HBase-1.1...m2je:4.x-HBase-1.1-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-HBase-1.2...m2je:4.x-HBase-1.2-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-HBase-1.3...m2je:4.x-HBase-1.3-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-HBase-1.4...m2je:4.x-HBase-1.4-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-cdh5.11...m2je:4.x-cdh5.11-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-cdh5.12...m2je:4.x-cdh5.12-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-cdh5.13...m2je:4.x-cdh5.13-PHOENIX-3547
https://github.com/apache/phoenix/compare/4.x-cdh5.14...m2je:4.x-cdh5.14-PHOENIX-3547

@m2je  Thank you for the contribution. I have committed this patch to the 4.x and master branches.
For future reference you only need to create a single PR. Once it is approved please follow instructions on how to generate a patch from the website (https://phoenix.apache.org/contributing.html), and attach the patches to the JIRA.

Thanks @twdsilva
Much appreciate your help. Will close this PR then.


why azure-storage is version -0.20.3?

I saw that azure-storage has already version 0.34.2, why if I pip install azure I get azure-storage==0.20.3
is that supposed to be like that?
Thanks

Hi @guydou
Yes, it's expected, please read the discussion about the stable azure package 1.0.3 and the preview version 2.0.0rc6:
http://azure-sdk-for-python.readthedocs.io/en/latest/installation.html
We plan to release to stable 2.0.0 soon, this will simplify the situation. Sorry for the inconvenience.

Hi @guydou
azure 2.0.0 is released, with version 0.34.2
Thanks again for the feedback!


Fixes deprecations for Swift 4.1

The flatMap call is deprecated in Swift 4.1 This code will check for the version and adjust the calls accordingly.

👍 👏


How to train on multi-GPUs when using fit_generator?

Hello guys,
I have opened a p3.8xlarge instance to benefit from its 4-GPUs but the training time improves only by a factor of 2 (x2) compared to my P3.2xlarge machine (not x4 and not even x3.. and I'm a bit disappointed)
I believe that it's due to the data_generator that slows down the process. Is there a way to overcome this issue? I have directories with thousands of images (~30GB) so I compelled to use flow_from_directory attribute.
Here's a sample from my code:

def get_batches(dirname, gen=image.ImageDataGenerator(), shuffle=True, batch_size=8, class_mode='categorical',
                target_size=(256,256), classes = None):
    return gen.flow_from_directory(dirname, target_size=target_size, classes=classes,
            class_mode=class_mode, shuffle=shuffle, batch_size=batch_size)

with tf.device('/cpu:0'):
    model = ResNet50(weights=None, include_top=True,
                     input_shape=(224, 224, 3),
                     classes=3)

parallel_model = multi_gpu_model(model, gpus=4)
parallel_model.compile(loss='categorical_crossentropy',
                       optimizer=Adam(), metrics=['accuracy'])

filepath = '/home/ubuntu/efs/images/IndexNew/toSep/KRAS_RN_multiGPU.h5'
checkpointer = ModelCheckpoint(filepath=filepath, verbose=1, save_best_only=True, save_weights_only=True)
stop_train = EarlyStopping(monitor='val_acc', patience=7, verbose=1)
reduce_lr = ReduceLROnPlateau(monitor='val_acc', factor=0.5,
            patience=2, min_lr=0.00001)

callbacks = [checkpointer, reduce_lr, stop_train]

path = '/home/ubuntu/efs/images/IndexNew/toSep/'
gen=image.ImageDataGenerator(horizontal_flip=True, vertical_flip=True)
batches = get_batches(path+'train', gen, batch_size=batch_size, shuffle=True, target_size=(224,224))
val_batches = get_batches(path+'valid', batch_size=batch_size, shuffle=False, target_size=(224,224))
parallel_model.fit_generator(batches, steps_per_epoch=batches.n//batch_size, epochs=20,
                    validation_data = val_batches, validation_steps = val_batches.n//batch_size,
                    verbose=1, callbacks=callbacks, workers=8, use_multiprocessing=True)

Thanks!

Correct me if I'm wrong, but if your batch_size=8, multiple GPU won't give you that much speedup right? It says here, https://keras.io/utils/#multi_gpu_model that,
if your batch_size is 64 and you use gpus=2, then we will divide the input into 2 sub-batches of 32 samples, process each sub-batch on one GPU, then return the full batch of 64 processed samples.

This induces quasi-linear speedup on up to 8 GPUs.

You're right but I'm actually using batch_size = 64*4
The default of my function is 8 but I change it when I call "get_batches"

max_queue_size: Integer. Maximum size for the generator queue. If unspecified, max_queue_size will default to 10.

max_queue_size in fit_generator(), default is 10. Is your model training faster than your generator generating batches?

@JeniaNovellusDx You got any solution for this? I just tried training a model on p2.8xlarge with 8 GPUs. I'm also not getting the required speedup! Theoretically, 1 epoch should have to take around ~3000s of time, but it's almost taking 6x more time! I'm keeping batch_size = 512*8 so that all 8 GPUs get batch of 512 for training, my max_queue_size = 300 so that I have enough data cached in memory to consumed by all GPUs. I'm using all workers=32 with thread safe generator to generate data from generator, I don't get it where the bottleneck is!? Any thoughts? I tried lr=1e-3 on Adam, still no improvement :(

What's # of workers? I believe that the more the better, but how should I know what precise #workers do I have

workers: Integer. Maximum number of processes to spin up when using process based threading.

New finding: Just spin up the p2.xlarge with 1 GPU, and 1 epoch is taking approx. 8:00:00 hours to finish, on the other hand, 8 GPUs are taking approx. 6:00:00 hours! This doesn't make any sense :(

@spate141 Any Updates regarding that ? I am facing the same issue in 1080 GTX 8 GB with tf 1.4.0 and  keras 2.1.3. I am using single GPU and getting the issue.

@mohapatras I don't think there is any issue with single GPU. You can basically get the speedup without making any changes in your code. If you somehow not getting boost, you can check how you are doing pre-process before feeding the data to your model. As I understand, most of the pre-processing is done on CPU and if you are using generator, disk i/o can be the main bottleneck.
P.S: if yo are using any methods/functions from Keras or other package in pre processing, you can write your own version with Numpy!

I am reading the data from disk using DataGenerator in Keras. I am simulating 256 x 256 images of total 56k in training and 2k in Val. It takes 6 hours/epoch which is insane.

if you are using generator, disk i/o can be the main bottleneck.

Any workarounds regarding this ?

Perhaps you can find some information here: #68, #107 or Python generator pre-fetch?

@mohapatras Hi, is your disk drive is HDD or SSD? If it is HDD, use_multiprocessing=True might worsen the performance of data input.

@mohapatras Can you share the number of input samples you are testing with?

@mohapatras @spate141 Hi, can you provide me the benchmark difference between use_multiprocessing=True and use_multiprocessing=False, because I just want to know whether you have the same issue.
From my experiments, I see no performance improved when use_multiprocessing=True:
use_multiprocessing=False, workers=1, data=disk, gpu=4, perf=88s/epoch
use_multiprocessing=True, workers=8, data=disk, gpu=4, perf=89s/epoch

@ghostplant Currently I'm not running the instance. My issue was solved by adjusting the way I was fetching data from disk with generator, and pre-processing it before feeding to GPU.

I also noticed the same behavior, If I use thread safe generator (#1638) with use_multiprocessing=True, workers=28 or 32, performance was worse actually compared to same thread safe generator but with use_multiprocessing=False.

From the nvidia-smi log, all 8 GPUs were ~70% usage when multiprocessing was off, on the other hand when I turned on multiprocessing, usage was oscillating between 0-40% on all 8 GPUs.

I will post exact log next time when I start the instance again.
Cheers!

Good news! So do you solve the bottleneck by putting some files into memory?

I launched new EC2 P3.8xlarge with the following packages:
Keras 2.1.3
tensorflow 1.5.0
cudnn 7
cuda 9
I also set the # workers to be 16 (cpu count // 2) use_multiprocessing=True and it seemed to perform well at 3.4x speedup compared to 1 gpu

Benchmark things separately always helps with understanding. You can just benchmark without worrying about the ImageGenerator part first, for example:
import keras
keras.backend.set_image_data_format('channels_first')
from keras.applications.resnet50 import ResNet50
from keras.utils import np_utils
import numpy as np

NUM_GPU = 1
batch_size = 32 * NUM_GPU

img_rows, img_cols = 224, 224

X_train = np.random.random((batch_size, 3, img_rows, img_cols)).astype('float32')
Y_train = np.random.random((batch_size,)).astype('int32')
Y_train = np_utils.to_categorical(Y_train, 1000)

def gen():
    while True:
        yield (X_train, Y_train)

model = ResNet50(weights=None, input_shape=X_train.shape[1:])

if NUM_GPU != 1:
    model = keras.utils.multi_gpu_model(model, gpus=NUM_GPU)

model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['accuracy'])
model.fit_generator(gen(), epochs=100, steps_per_epoch=50)

@ghostplant Yes, I did converted data in format which can be loaded in memory and then with little post-processing I was managed to feed data to model on 8GPUs, all 8 GPUs usage was ~70-75% so I can it pretty much was the disk i/o in my case.

@ppwwyyxx Hi, thx for your example and could i discuss something about the model defining?
In your example, you defined the model on gpu, not like keras doc's suggested on cpu:
model = ResNet50(weights=None, input_shape=X_train.shape[1:])
if NUM_GPU != 1:
    model = keras.utils.multi_gpu_model(model, gpus=NUM_GPU)

Recent days, I found that after added the cpu scope with with tf.device("/cpu:0"):,  something doesn't make sense appeared is that my gpu didn't working for training :( . So did you ever have faced this problem ?
@ghostplant Also, did you test the benchmark of with and without cpu scope?
Thx everybody, it confused me a lot. And my env is as follows:

tensorflow = 1.7.0
keras = tf.keras

You certainly should not put model on CPU if you want to use a GPU.

@Hayao41 I did both kinds of benchmarks, and GPUs are always working. However, the performance of using tf.device('/cpu':0) is a bit faster.

@ppwwyyxx @ghostplant thx guys, but it doesn't work for me :( as above. If I drop the cpu scope, all things go to normal, working hard.
 I used keras mainstream version installed from pip.

@ghostplant This may be a spot, I will go to have a test on mainstream. Thx very much!

I am using tf.keras and tf.keras.sequence to load data. I also rewrite my code in pytorch. I found it's much slower when using keras. I want to kown if it's the reason that torch.utils.data.Dataset is running faster than tf.keras.sequence

@xuzheyuan624 Why keras multi-gpu is slow:

Each GPU should have a self-owned data generators;
data generator should store image inputs to memories allocated by cuMemHostAlloc();
Allreduce should be used instead of using parameter-server;

If the above 3 problem solved, keras multi_gpu could have as fast performance as the version of official Tensorflow CNN benchmark.

@ghostplant So just use keras.utils.multi_gpu_modeland set use_multiprocessing=True when using fit_generator is not enough to run fast ? Are there any examples to solve this problem?

@xuzheyuan624 Yes, it is just a a work around that looks simple. If using the fastest solution, you have to modify lots of codes related to input processing and gradient updates.

@ghostplant OK, it's sound too difficult for me to speed up the code with keras. I will train my model on pytorch and then transform it's weights to keras.

You can also use tensorpack which includes built-in fast multi-GPU solution. Some comparison scripts here.

Hey Guys!
Just found the solution and it has been running properly haha!
We cannot use the traditional self-made generator as the GPU may have varying end-training times and may send the information to one model for same batch sizes. Therefore become NOT thread-safe. The generator needs to be sequenced, Hence, thread-safe (tf.keras.utils).
I am running my model on g3-16xlarge due to the size of the model and the size of the training set!
The generator is now a class and not a method!
`
    class traingen(tf.keras.utils.Sequence):

       def __init__(self, batchSize):
           self.dataset = pf.read_csv('mydataset.csv')
           self.batchSize = batchSize

       def __len__(self):
           return self.dataset.shape[0] // self.batchSize

       def getLen(self):
           return self.dataset.shape[0] // self.batchSize

       def __getitem__(self, idx):
           rows = random.sample(range(0, self.dataset.shape[0]), batchSize)
           return self.dataset[rows]

`
The following code should work when inserted like
          history = model.fit(           traingen,                                    epochs = 5,                                    steps_per_epoch = trainSeq.getLen(),                                    verbose = 1,                                    use_multiprocessing = True,                                    workers = 32          )
This works on 4 GPUs parallel training!
Good Luck yall!

Hi, I went through all the discussions and the outcome I got is that you need to use a data generator based on the keras.utils.Sequence within the fit_generator, setup the batch size divisible by the number of GPUs available and go for the use_multiprocessing = True. This didn't work in my case. I'm training the system for NLP sequence-to-sequence method. The data is sentences (text) with source and target. Multiprocessing on 4 GPUs is still 50% for that of a single GPU. Something is wrong and from my experience it's related to keras. I'm using TensforFlow GPU v1.15.3 (latest before v2.0), and keras v2.1.5 (tried newer ones, no change).
I have no reported errors nor warnings! I have 20+ years experience in programming / A.I. and I can say that this Python based framework is lacking, not easy to work with and makes things hard. Documentation is poor, and they keep rolling out new versions "deprecating" former methods and classes.
Based, on Amdahl's law, there should be some noticeable speedup as this task is for sure paralellizable. I believe it's a keras issue ..


Long tags with alias/translation not fully visible

It's still covered, maybe covered by the tag count?

Interesting, it only seems to happen on Firefox. It probably handles some part of the CSS differently, I'll have a look.

Found the issue, it seems that Firefox by default doesn't include the scrollbar in its width/text overflow calculation, causing it to clip despite having no maximum width. After explicitly setting scrollbar-gutter to stable in the CSS, it seems to work. Will push in a minute.


port fingerprint docs from 0.4.x

adds some minor tweaks as well
/cc @lukemelia (sorry for all the pings)

Can we verify that the syntax in this example works? I think my ember-cli-build.js file gets the environment differently.

was about to say... it's on my list for sure, will try to get to it tomorrow


⚠ Young Discoverers has degraded performance

In 8cbec57, Young Discoverers (https://youngdiscoverers.org.au) experienced degraded performance:

HTTP code: 200
Response time: 5984 ms

Resolved: Young Discoverers performance has improved in 51212fd.


fix(executor): wiped postate storage

Potentially, fixes #1848. Keeping as a draft for now

Codecov Report

Merging #1849 (f8544cd) into main (a59806f) will decrease coverage by 55.81%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff             @@
##             main    #1849       +/-   ##
===========================================
- Coverage   73.40%   17.60%   -55.81%
===========================================
  Files         415      415
  Lines       50555    50555
===========================================
- Hits        37110     8898    -28212
- Misses      13445    41657    +28212

Flag
Coverage Δ

integration-tests
17.60% <0.00%> (-2.11%)
:arrow_down:

unit-tests
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

crates/executor/src/executor.rs
0.00% <0.00%> (-90.44%)
:arrow_down:

crates/executor/src/substate.rs
0.00% <0.00%> (-16.33%)
:arrow_down:

crates/storage/provider/src/post_state.rs
0.00% <0.00%> (-89.98%)
:arrow_down:

... and 294 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Tested it and doesn't seem to fix the issue. @onbjerg pls confirm this is a bug

@gakonst i'll keep this PR as draft until u finish resyncing


fixed a display bug within the slow and running queries view

This needs to be merged into devel as well.


Document string support

Documentation for preliminary string support

Codecov Report

Merging #3506 into master will increase coverage by 0.06%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3506      +/-   ##
==========================================
+ Coverage   80.62%   80.69%   +0.06%
==========================================
  Files         392      392
  Lines       79770    80130     +360
  Branches     9063     9117      +54
==========================================
+ Hits        64317    64663     +346
- Misses      14038    14051      +13
- Partials     1415     1416       +1

OK, I have accepted the proposed change.

hmmm, that edit functionality on github doesn't line wrap or deal with white space, guess that's expected, so apologies, this needs manual fixing now. The change renders ok as HTML (apart from a space before the full stop), but the source lines overrun. Perhaps edit manually?! Guess we've learned not to do that again for RST.


Registry configuration

The splunk-as-a-service-app is trying to pull images from the internet, which is not possible for air gapped deployments. I need a way to specify the server to pull from.

Commit c36d40a7b50ec6d12ebcc292927ed81b5d6a087f closes this. In the cluster settings, you can now specify the Splunk image to use. That way you can specify which Splunk version to use, but also specify the image repository to get the image from. Full format hostname[:port]/username/reponame[:tag]


python311Packages.maison: 1.4.2 -> 1.4.3

Diff: https://github.com/dbatten5/maison/compare/refs/tags/v1.4.2...v1.4.3
Changelog: https://github.com/dbatten5/maison/releases/tag/v1.4.3
Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Result of nixpkgs-review pr 281188 run on x86_64-linux 1

4 packages failed to build:

yamlfix (python311Packages.yamlfix)
yamlfix.dist (python311Packages.yamlfix.dist)
python312Packages.yamlfix
python312Packages.yamlfix.dist

4 packages built:

python311Packages.maison
python311Packages.maison.dist
python312Packages.maison
python312Packages.maison.dist

I'll have a look tonight, unless someone else beats me to it.

This seems to have broken yamlfix now.
error: builder for '/nix/store/0aplk046a15280sdaqlq03561ydb755c-python3.11-yamlfix-1.16.0.drv' failed with exit code 1;
       last 10 log lines:
       > E                 File "/nix/store/asiphbpiy2gmidfm3xbwcikayhs66289-python3-3.11.7/lib/python3.11/re/_parser.py", line 170, in __getitem__
       > E                   return SubPattern(self.state, self.data[index])
       > E                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
       > E               ResourceWarning: unclosed file <_io.FileIO name='/build/pytest-of-nixbld/pytest-0/popen-gw0/test_check_one_file_changes0/source.yaml' mode='rb+' closefd=True>
       >
       > /nix/store/xbsrnrzlv658h5r85pkw6v17ns63wlnx-python3.11-pytest-7.4.3/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
       > =========================== short test summary info ============================
       > FAILED tests/unit/test_services.py::TestFixCode::test_fix_code_respects_parent_lists_with_comments - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO ...
       > ======================== 1 failed, 106 passed in 2.81s =========================
       > /nix/store/v5irq7wvkr7kih0hhnch5nnv2dcq8c4f-stdenv-linux/setup: line 1557: pop_var_context: head of shell_variables not a function context
       For full logs, run 'nix log /nix/store/0aplk046a15280sdaqlq03561ydb755c-python3.11-yamlfix-1.16.0.drv'.


fix: home token balance sync

Description
Keeps the total balance in sync with the transaction feed sans pending transactions.
Other changes
Snapshots updated
Video
https://user-images.githubusercontent.com/26950305/187009176-83d8cd2e-96b2-4318-977b-c0ff3ebf81b7.mov
Tested
Tested on an iPhone 13 on Mainnet.
How others should test
Prerequisites: Using Mainnet

fund a new wallet - see that total balance updates when receiving funds.
Send funds to another wallet and obverse that total balance updates. It can take up to ~13 seconds.

Related issues

Fixes: RET-322, #2578

Backwards compatibility
Yes

@kathaypacific adding yield takeEvery(Actions.UPDATE_TRANSACTIONS, refreshBalances) in watchNewFeedTransactions generates a lot of network requests that are likely unneeded such as polling the exchange rate. I'll take a look and see if there is another way besides keeping track of the last transaction hash in the component state.

@kathaypacific just need to implement a schema migration to handle the change of loading to showLoading. Should that be included in this PR or should I revert the name change and do that in a follow up PR?

@kathaypacific I used batch so we don't get multiple rerenders and because it's simpler than handling it in the Saga's. I haven't seen any issues with in testing, but I'll keep my eyes for any issues that arise during @ValoraQA testing.

Hey @MuckT we have verified the above issue on Latest iOS Test Flight Release build V 1.40.0 and updated status in test rails
Observation :

User is able to see the total balance updated within 13 seconds while sending and receiving funds to another user

Test Rails:
iOS 15 : https://valoraapp.testrail.io/index.php?/tests/view/69495
Device : iPhone 13 mini(15.1.1)
Thanks..!

Hey @MuckT we have verified the above task on Latest Android Internal Release build V 1.40.0 and updated status in test rails
Test Rails:
Android 11: https://valoraapp.testrail.io/index.php?/tests/view/69557
Device : Google Pixel 2XL(11.0)
Thanks..!


the trait embedded_hal_async::i2c::I2c is not implemented for Generic<embedded_hal_mock::eh1::i2c::Transaction>

Hello, thank you for this crate, it's awesome! I'm having an issue with mocking embedded_hal_async::i2c::I2c. Is there support for this?

I looked deeper into the source and see that it isn't [yet] so I added support for it.


Use double instead of real_t type for time-related parameters and variables

This is a draft pull request that addresses #65313 in that time-related variables and parameters types have been changed to use double type.
I changed everything I found except the Animation class and some Tween/Tweener types.
I left the Animation class alone as I ignore how much of its data is type delicate.
Tween/Tweener interpolators relies heavily on real_t values. I think it should be using double instead.

Are you aware of the cicd github actions error. I support this work!

I am aware, thanks for the heads up!

Looks good! Could you squash the commits? See PR workflow for instructions.

@akien-mga done
thanks everyone

Thanks!


Ubuntu support

Rat check failing for following files:
- .../deploy/DEB/templates
- .../deploy/DEB/config

Thanks for the comments, Rahul. I am currently testing the new changes, no need to test these temporary commits.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/635/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/636/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/637/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/638/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/639/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/640/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/641/

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/madlib-pr-build/643/


Fix the path of the map database

The postmap command needs the complete path of the database. In the exec the $name was used, while it needs $path.

Thanks @amateo!


Add an RPC operation, "getDefaultViewStateData" to minimize back and forth communication between frontends and backends

The aim of this PR is to minimize the amount of back and forth communication between our frontends and backends when it comes to generating a view state. This is done by moving some code from the frontend to the backend and putting it behind a more specific RPC endpoint, "getDefaultViewStateData". Previously there were many different operations called in order to get this data.

Do you intend to do anything about ViewCreator2d?

Do you intend to do anything about ViewCreator2d?

I'll defer to @wgoehrig here...

Do you intend to do anything about ViewCreator2d?

I'm open to considering it later, but our priority right now was to make sure there's a "hot path" for iTwin Viewer, which I believe is only using ViewCreator3d... @kckst8 can you confirm?
@pmconne do you have a preference here?  Would you rather a more complicated RPC method if it meant it could handle both 2D and 3D?  Or two separate RPC methods?

Would you rather a more complicated RPC method if it meant it could handle both 2D and 3D? Or two separate RPC methods?

Separate RPC methods probably better. But type/function names should include "3d" or "2d" in their names.


NaN values during the optimization using ego optimizer in Rust

I am using egobox/ego optimizar to minimize an object function, optimizer logs are like the following:
Best fun(x)=[20.1200008392334] at x=[21.101404362038778, 17.098729966743218, 13.740734782832677]
[2023-05-29T07:21:39Z INFO  egobox_ego::egor_solver] Train surrogates with 126 points...
[2023-05-29T07:21:43Z WARN  egobox_ego::egor_solver] Criterion scale computation warning: (1493 NaN + 0 Inf) / 2600 points
[2023-05-29T07:21:43Z INFO  egobox_ego::egor_solver] Infill criterion scaling is updated to 60238942.50300943
[2023-05-29T07:21:43Z INFO  egobox_ego::egor_solver] Optimize infill criterion...
[2023-05-29T07:21:43Z INFO  egobox_ego::egor_solver] +1 point(s), total: 127 points

my question is about the warning that appears; What are NaNs and how can i resolve that?

The cost function that Im using:
The size of input space is 1x24, and I only want to first 3 indices to be close to the desired target.
for ind in 0..x.shape()[0]{  let optimization_error = (x[[ind, 0]] as f32 - 5.0).pow(2.0)                                              + (x[[ind, 1]] as f32 - 10.0).pow(2.0)                                             + (x[[ind, 2]] as f32 - 6.0).pow(2.0); }
the initial logs are as follows:
running 1 test
[2023-05-29T18:21:10Z INFO  egobox_ego::egor_solver] Compute initial LHS with 27 points
[2023-05-29T18:21:10Z INFO  egobox_ego::egor_solver] Train surrogates with 27 points...
[2023-05-29T18:21:10Z INFO  egobox_ego::egor_solver] Objective initial clustering and training...
[2023-05-29T18:21:10Z INFO  egobox_ego::egor_solver] ... Objective trained (1 / Mixture[Smooth(1)])
[2023-05-29T18:21:11Z WARN  egobox_ego::egor_solver] Criterion scale computation warning: (994 NaN + 0 Inf) / 2600 points
[2023-05-29T18:21:11Z INFO  egobox_ego::egor_solver] Infill criterion scaling is updated to<PHONE_NUMBER>013367
[2023-05-29T18:21:11Z INFO  egobox_ego::egor_solver] Optimize infill criterion...
[2023-05-29T18:21:11Z INFO  egobox_ego::egor_solver] +1 point(s), total: 28 points
[2023-05-29T18:21:11Z INFO  egobox_ego::egor_solver] ********* End iteration 1/100: Best fun(x)=[3.956397771835327] at x=[6.196287066574575, 9.960299900850034, 7.588621562259139, 5.07007657420999, 15.189891199076364, 3.3795400897239123, 9.95242390791087, 13.28904490220349, 19.16940303541127, 6.446226029095112, 13.867988241190531, 6.50821667597554, 7.779694029683958, 11.334380960131215, 1.368792894200031, 11.748846313543726, 22.37531483493761, 8.833127960775252, 18.335932042837978, 17.678943680826798, 13.515041716316743, 4.213578394250486, 15.213664448356239, 3.154915644666341, 12.796106854109512, 10.663754421782311]
full logs can be found here:
https://ufile.io/h2a3ncw1

Hmm... I am not sure to understand. You have changed your objective function? In your first logs x was 3-dimensional, now it is 24-dimensional. Could you show me your code ? How do you build and run Egor (res = EgorBuilder::optimize...) ?

I experimented lots of examples to reproduce the same Warning, and yes I changed the cost function. The last one that i tried has 26 dim. I found that it is related to the xlimits (x upper and lower bounds). In case that I set limits for the first 23 dims, I saw No warnings
running 1 test [2023-05-30T19:43:33Z INFO  egobox_ego::egor_solver] Compute initial LHS with 27 points [2023-05-30T19:43:33Z INFO  egobox_ego::egor_solver] Train surrogates with 27 points... [2023-05-30T19:43:33Z INFO  egobox_ego::egor_solver] Objective initial clustering and training... [2023-05-30T19:43:33Z INFO  egobox_ego::egor_solver] ... Objective trained (1 / Mixture[Smooth(1)]) [2023-05-30T19:43:34Z INFO  egobox_ego::egor_solver] Infill criterion scaling is updated to<PHONE_NUMBER>798867 [2023-05-30T19:43:34Z INFO  egobox_ego::egor_solver] Optimize infill criterion... [2023-05-30T19:43:34Z INFO  egobox_ego::egor_solver] +1 point(s), total: 28 points [2023-05-30T19:43:34Z INFO  egobox_ego::egor_solver] ********* End iteration 1/5: Best fun(x)=[1.6070678803847929] at x=[6.098548252442424, 10.616900950952962, 6.140331158884331, 1.9170139613874455, 22.602510909025227, 17.502471725128004, 10.66970007420761, 2.4841483927907375, 14.685108105439781, 20.099759664828714, 6.249214379178304, 5.130267352261458, 16.11728224671063, 20.09779580435667, 16.452331536070183, 9.130985640951744, 7.263912165024908, 16.28346793549735, 4.776299954473808, 7.838135225491183, 11.94484073546002, 11.045370670465292, 14.255468720558094, 0, 0, 0]
However adding limit to one more dimension, caused the warning to show up:
running 1 test [2023-05-30T19:46:06Z INFO  egobox_ego::egor_solver] Compute initial LHS with 27 points [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] Train surrogates with 27 points... [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] Objective initial clustering and training... [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] ... Objective trained (1 / Mixture[Smooth(1)]) [2023-05-30T19:46:07Z WARN  egobox_ego::egor_solver] Criterion scale computation warning: (336 NaN + 0 Inf) / 2600 points [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] Infill criterion scaling is updated to<PHONE_NUMBER>160128 [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] Optimize infill criterion... [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] +1 point(s), total: 28 points [2023-05-30T19:46:07Z INFO  egobox_ego::egor_solver] ********* End iteration 1/5: Best fun(x)=[0.8971263111591025] at x=[5.8890160307073955, 10.274799515985617, 6.176810730212961, 14.495103127624956, 15.30528925673276, 6.155674604969699, 7.188728900217343, 12.906229066791356, 13.65068438189934, 20.72991466177427, 22.58180777411574, 5.112200410088319, 9.852378885666893, 20.49066874421727, 21.248352906097274, 9.999891644271038, 20.38340905382666, 5.208168134154029, 18.4837423872606, 10.572670777494782, 1.0887189391429617, 19.2325132632352, 15.425764074849422, 10.46127440110147, 0, 0]
and yes, The cost function is still the same, and the call part is :
    let mut xlimits = Array2::<f64>::zeros((operation_time_boundary.len(), 2));     xlimits[[0, 0]] = 1.0;     xlimits[[0, 1]] = 24.0;     .     .     .     let optim_results = EgorBuilder::optimize(cost_fn)         .min_within(&xlimits)         .n_iter(max_iter)         .target(1e-1)         .run()         .expect("Minimize failure");

The warnings may indicate the optimizer has a hard time approximating your objective function.
From the optimizer standpoint the number of rows of xlimits gives the dimension of x (the lower the better!), hence by increasing xlimits you are increasing the dimension of the problem. By default we try to call as few as possible the objective function (in case the evaluation is expensive), we start with an initial set of evaluations (the doe) of (dimension of x) + 1 points.
If your objective function is not that expensive to evaluate you can help the optimizer by starting with more points, like 3 * nx where nx is the dimension of x (ie the number of rows of xlimits). To set the size of this initial doe use the n_doe() method, something like:
let optim_results = EgorBuilder::optimize(cost_fn)
    .min_within(&xlimits)
    .n_iter(max_iter)
    .n_doe(3 * xlimits.nrows())
    .target(1e-1)
    .run()
    .expect("Minimize failure");


Looks like «_has_package» from «pyinfra/operations/util/packaging.py» works wrong

Describe the bug
I catched situation with missing DNF package after pyinfra 2.6 operation.
dnf.packages(
    name="Test packages RPMs",
    packages=["caddy"],
    _sudo=True,
)

I am not sure, that I can easily provide convinient reproducable environment (publicly available virtual machine in some cloud? OVA/qcow fromm virtualbox/qemu/docker…).
But I did some debugging, and I see strange behaviour, probable suspect.
def _has_package(package, packages, expand_package_fact=None, match_any=False):

definitely should return True only if package is installed (it exists exists in packages).
But last lines of this function return True even if no such package exists, because all([])==True.
…
    if match_any:
        return any(checks), package_name_to_versions
    return all(checks), package_name_to_versions

Illustrative scheme from my debugging

I hope, this scheme can describe the issue…
If not, I will try to make reproduceable environment, but not sure, how much time it takes if  I can succeed even.
Meta

Include output of pyinfra --support.

--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-6.0.15-200.fc36.x86_64-x86_64-with-glibc2.35
      Release: 6.0.15-200.fc36.x86_64
      Machine: x86_64
    pyinfra: v2.6
    Executable: /usr/local/bin/pyinfra
    Python: 3.10.9 (CPython, GCC 12.2.1 20221121 (Red Hat 12.2.1-4))

How was pyinfra installed (source/pip)?
pip

Hi @belonesox, thank you for reporting this! I am unable to replicate it currently, a reproducible environment would be extremely helpful (docker if possible).
Following the code it seems if package_name_to_versions is non-empty the checks should always contain values. Might be worth converting checks to a list rather than a generator for testing here too.

No repro, please re-open if you have/get one.


Azure deployment source config - changing branch doesn't seem to work

Description
I have a webapp where I already configure my git deployment integration. Now I want to quickly change the branch I'm using but it seems that the branch never changes.
I'm running
az webapp deployment source config --repo-url MY_REPO_URL --branch staging --name MY_APP --resource-group MY_RESOURCE

The output seems ok: Finished and shows the right branch. Then I run the sync command but it seems the branch isn't really changed. Checking my site\deployments\settings.xml file using KUDU also shows old branch is still there. Basically branch didn't change.
Seems to me like a bug but please advise if I should be running something else. In CLI 1.0, in theory we had a command to only change branches so that's a good thing to have also in CLI 2.0

Environment summary
Install Method: How did you install the CLI? (e.g. pip, interactive script, apt-get, Docker, MSI, nightly)
MSI
CLI Version: What version of the CLI and modules are installed? (Use az --version)
azure-cli (2.0.7)
OS Version: What OS and version are you using?
Windows 7
Shell Type: What shell are you using? (e.g. bash, cmd.exe, Bash on Windows)
cmd.exe

I will fix this in CLI. The root cause is the another webapp service api weirdness, that no error was reported when it was not able to finish off the work.

@panchagnula can you please check if the work around is working, if yes let's close this issue

//CC:@mayurid
We probably should leave these issues along, which are tagged with "ServiceAttn", as someone from SDK team will reach to your team in future for possible fixes at the service end.

Closing this since this is old & from the previous notes not something we will fix on the client side. Please open a new issue if this is still a problem. Thank you,


REFACTOR: VueJS Router file to incorporate 'children' tag

This should allow better management of routes.

Archiving project. Client decided to remove vue/vite and bulma frameworks.


fix changes in functional tests workflow

Goal

Testing

Release Notes

WHAT:
WHY:

#1005  👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @nelsitoPuglisi and the rest of your teammates on  Graphite


CI: [Travis] github.com/cilium/cilium/daemon/cmd  panic: send on closed channel

See in https://app.travis-ci.com/github/cilium/cilium/jobs/586349585
goroutine 675 [running]:
github.com/cilium/cilium/pkg/identity/cache.(*localIdentityCache).release(0x400097f400, 0x40009aa300)
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/identity/cache/local.go:139 +0x148
github.com/cilium/cilium/pkg/identity/cache.(*CachingIdentityAllocator).Release(0x40001eaf60, {0x3159220, 0x4000757140}, 0x40009aa300, 0x0?)
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/identity/cache/allocator.go:427 +0x2bc
github.com/cilium/cilium/pkg/ipcache.(*IPCache).releaseCIDRIdentities(0x40003c3280, {0x3159220, 0x4000757140}, {0x40009aa4c0, 0x2, 0x400105f1f0?})
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/ipcache/cidr.go:220 +0x208
github.com/cilium/cilium/pkg/ipcache.(*asyncPrefixReleaser).run(0x40000c9130, {0x3159220, 0x4000757140}, {0x400105f1e0, 0x1, 0x1})
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/ipcache/gc.go:103 +0x1ac
github.com/cilium/cilium/pkg/ipcache.newAsyncPrefixReleaser.func1({0x400105f1e0, 0x1, 0x1})
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/ipcache/gc.go:53 +0x84
github.com/cilium/cilium/pkg/trigger.(*Trigger).waiter(0x40000fed20)
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/trigger/trigger.go:201 +0x45c
created by github.com/cilium/cilium/pkg/trigger.NewTrigger
	/home/travis/gopath/src/github.com/cilium/cilium/pkg/trigger/trigger.go:122 +0x1b8

@joestringer This seems to have been caused by https://github.com/cilium/cilium/commit/4afbcc38ce91118f1cb4a9f715a268775cdf7f6e

:+1: initial triage is that the unit tests shut down the identity allocator and the ipcache in the wrong order that can trigger this race condition / crash. The identity allocator is not currently integrated into the main Daemon.Close() function so this only impacts the testsuite. We should be able to move identtiy allocator cleanup into that function.


JFXCentral Badge

Would be great if you could add this badge to your readme.
[![JFXCentral](https://img.shields.io/badge/Find_me_on-JFXCentral-blue?logo=googlechrome&logoColor=white)](https://www.jfx-central.com/libraries/richtextfx)

Done, thanks Dirk :-)


Support to ember-engines

Hi @chriseppstein I'm working o roadmap to ember-engines 1.0. I just don't understand how this does not work with ember-engines, and Linkedin uses ember-engines on its main site. Maybe it's a workaround, I don't know
But I would help on anything to make this able of running with ember engines.
thanks!

@villander I just tried css-blocks with ember-engines.
The way it works is that the CSS is included in the app.css file instead of the engine's. So in that sense, this isn't working as expected.

@GCheung55 The optimization step requires all of the styles be optimized together. Once that is done, the styles can then be split back into their respective engines. Code splitting is a feature we plan to add later. The feeling right now is that a global optimization may pay enough dividends that code splitting isn't a very big win. As with most things performance related, the answer is probably "it depends". Either way, we plan to implement the code-splitting feature in the future.


fix: bug with prs from forks returning incorrect set of changed files

Closes: #714

@jackton1  Which version should we use to get this fix?

@jackton1 Which version should we use to get this fix?


Error received in the gateway container whilst trying to connect.

Simulator returns the following error whilst trying to connect.
ERROR:root:CP1: received 1011 (internal error); then sent 1011 (internal error)
Gateway Container is showing the following Log output.
| Successfully set IOT certificates |
| Starting the gateway server |
| We renamed asyncio-mqtt to aiomqtt and released a version 1.0.0 in the process. This is the last release under the asyncio-mqtt name. You can find the new repository at https://github.com/sbtinstruments/aiomqtt |
| Exception ignored in: <function Client.del at 0xffff92890040> |
| Traceback (most recent call last): |
| File "/opt/ocpp-gateway/lib64/python3.10/site-packages/paho/mqtt/client.py", line 874, in del |
| self._reset_sockets() |
| File "/opt/ocpp-gateway/lib64/python3.10/site-packages/paho/mqtt/client.py", line 1133, in _reset_sockets |
| self._sock_close() |
| File "/opt/ocpp-gateway/lib64/python3.10/site-packages/paho/mqtt/client.py", line 1119, in _sock_close |
| if not self._sock: |
| AttributeError: 'Client' object has no attribute '_sock' |
| ERROR:websockets.server:connection handler failed |
| Traceback (most recent call last): |
| File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 236, in handler |
| await self.ws_handler(self) |
| File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 1175, in _ws_handler |
| return await cast( |
| File "/opt/ocpp-gateway/server.py", line 33, in handler |
| async with gateway.Gateway(charge_point_id, websocket) as iot_connection: |
| File "/opt/ocpp-gateway/gateway.py", line 37, in init |
| super().init( |
| File "/opt/ocpp-gateway/lib64/python3.10/site-packages/asyncio_mqtt/client.py", line 298, in init |
| self._client: mqtt.Client = mqtt.Client( |
| TypeError: Client.init() missing 1 required positional argument: 'callback_api_version' |

Exact same issue encountered.

Please try updating src/ocpp-gateway-container/requirements.txt to fix the version of the mqtt package to one that pre-dates the version increase of path-mqtt to v2.0.0; e.g.:
asyncio-mqtt==0.16.1
If that solves the problem we’ll update the same in the repo. Thx!

I've updated the src/ocpp-gateway-container/requirements.txt file and now get the following error in the Gateway Container logs:
Successfully set IOT certificates Starting the gateway server ERROR:websockets.server:connection handler failed Traceback (most recent call last):   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 236, in handler     await self.ws_handler(self)   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 1175, in _ws_handler     return await cast(   File "/opt/ocpp-gateway/server.py", line 33, in handler     async with gateway.Gateway(charge_point_id, websocket) as iot_connection:   File "/opt/ocpp-gateway/gateway.py", line 37, in __init__     super().__init__(   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/asyncio_mqtt/client.py", line 343, in __init__     self._client.message_retry_set(message_retry_set) AttributeError: 'Client' object has no attribute 'message_retry_set' ERROR:websockets.server:connection handler failed Traceback (most recent call last):   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 236, in handler     await self.ws_handler(self)   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 1175, in _ws_handler     return await cast(   File "/opt/ocpp-gateway/server.py", line 33, in handler     async with gateway.Gateway(charge_point_id, websocket) as iot_connection:   File "/opt/ocpp-gateway/gateway.py", line 37, in __init__     super().__init__(   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/asyncio_mqtt/client.py", line 343, in __init__     self._client.message_retry_set(message_retry_set) AttributeError: 'Client' object has no attribute 'message_retry_set' ERROR:websockets.server:connection handler failed Traceback (most recent call last):   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 236, in handler     await self.ws_handler(self)   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 1175, in _ws_handler     return await cast(   File "/opt/ocpp-gateway/server.py", line 33, in handler     async with gateway.Gateway(charge_point_id, websocket) as iot_connection:   File "/opt/ocpp-gateway/gateway.py", line 37, in __init__     super().__init__(   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/asyncio_mqtt/client.py", line 343, in __init__     self._client.message_retry_set(message_retry_set) AttributeError: 'Client' object has no attribute 'message_retry_set' ERROR:websockets.server:connection handler failed Traceback (most recent call last):   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 236, in handler     await self.ws_handler(self)   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/websockets/legacy/server.py", line 1175, in _ws_handler     return await cast(   File "/opt/ocpp-gateway/server.py", line 33, in handler     async with gateway.Gateway(charge_point_id, websocket) as iot_connection:   File "/opt/ocpp-gateway/gateway.py", line 37, in __init__     super().__init__(   File "/opt/ocpp-gateway/lib64/python3.10/site-packages/asyncio_mqtt/client.py", line 343, in __init__     self._client.message_retry_set(message_retry_set) AttributeError: 'Client' object has no attribute 'message_retry_set'

It looks like asyncio-mqtt use paho-mqtt for mqtt client. paho-mqtt client latest version is 2.1.0 but they removed message_retry_set at version 2.0.0.
https://github.com/eclipse/paho.mqtt.python/releases/tag/v2.0.0
So specify paho-mqtt version in src/ocpp-gateway-container/requirements.txt fixed this problem.
boto3
websockets
paho-mqtt==1.6.1
asyncio-mqtt

I'm having the same issue https://github.com/aws-solutions-library-samples/guidance-for-modernizing-electric-vehicle-charging-on-aws/issues/153.
It was not resolved.


Change Default Login Shell

Currently there's no way to change the login shell, that i'm aware of, so i'm currently stuck in zsh, but i'd like to be using bash.

Duplicate of #81.

You can check this out: https://youtu.be/fs7yWlB2dSU


LayoutLMv2, what kind of heuristics are used to locate the entity spans of the normalized answers for kleister-nda dataset?

Describe
Model I am using: LayoutLMv2
In the paper of LayoutLMv2, it says "We use heuristics to locate entity spans because the normalized standard answers may not appear in the utterance". What kind of heuristics are used?

The kleister-nda dataset contains very long sentences, how to deal with such long sentences?


Update Box.lastModified fields when changing box state during transfer

 2.0 BE Bug lastModifiedOn does not update when Box is assigned or unassigned in BoxesView


Add GetScopeName(id) method to ScopeIdProvider

Every id represents a group, while every group
needs a name to be shown under Live tab.
This adds the needed functionality to
ScopeIdProvider.
Tests: Unit
Bug: http://b/226889590

There is one commit called "scope names WIP". This is probably a mistake. Im writing this comment as a reminder to clean this up

@antonrohr yeah, I always clean the commit messages on squash.


Disable Scrutinizer; enable Codecov

This is the same as https://github.com/getsentry/sentry-php/pull/980

It seems to work: https://codecov.io/gh/getsentry/sentry-symfony/branch/codecov
Removing WIP


Can't use prompt-confirm

Hi,
Sorry it's me again.
I don't know if this is linked or not (because I had not used prompt-confirm before), but since I pulled the latest version of enquirer, to solve enquirer/enquirer#6, I cannot use the prompt-confirm.
For information (maybe this is useful) I have git pulled the enquirer project, then I cded to my workspace and did npm install /path/to/enquirer/clone, it installed and I could use enquirer.
Now I am trying to use prompt-confirm and this fails, here is a minimal snippet that fails on me:
'use strict';

const Enquirer = require('enquirer')
const inspect = require ('util').inspect

function query (i) {
    let enquirer = new Enquirer ()

    enquirer.register ('confirm', require ('enquirer-prompt-confirm'))

    enquirer.question ('QuestionA', 'Question A (' + i + ')?', {
		type: 'confirm'
	})
    enquirer.question ('QuestionB', 'Question B (' + i + ')?', {
        type: 'confirme',
    })

    enquirer.ask()
    .then (answers => {
        console.log ('Answers #' + i)
        console.log (inspect (answers))
        console.log ('--')

        if (i > 0) {
            console.log ('Asking for argument ' + i + ' now...')
            query (i-1)
        }
    })
	.catch( err => {
		console.error ('Err: ' + err)
	})
}

query (3)

As you noticed, I have used the same template to chain prompt-confirm commands, but it fails at the first: it asks me the first question, but then when I answer it fails. Here is the console session:
? Question A (3)? Yesreadline.js:982
            throw err;
            ^

TypeError: this.ui.write is not a function
    at Confirm.onSubmit (/home/nschoe/workspace/xxx/node_modules/enquirer-prompt-confirm/index.js:71:11)
    at UI.on (/home/nschoe/workspace/xxx/node_modules/component-emitter/index.js:65:8)
    at UI.Emitter.emit (/home/nschoe/workspace/xxx/node_modules/component-emitter/index.js:133:20)
    at emitOne (events.js:96:13)
    at Interface.emit (events.js:188:7)
    at Interface._onLine (readline.js:239:10)
    at Interface._line (readline.js:585:8)
    at Interface._ttyWrite (readline.js:862:14)
    at ReadStream.onkeypress (readline.js:126:10)
    at emitTwo (events.js:111:20)
    at ReadStream.emit (events.js:191:7)
    at emitKeys (internal/readline.js:389:14)
    at next (native)
    at ReadStream.onData (readline.js:972:36)
    at emitOne (events.js:96:13)
    at ReadStream.emit (events.js:188:7)

This is weird, so might it be a problem induced by the fix on enquirer? Or is it a cache or dependency error on my side?

Or is it a cache or dependency error on my side?

I might be either, but it's easiest to test the latter first. can you try deleting node_modules and reinstalling?

enquirer.register ('confirm', require ('enquirer-prompt-confirm'))

enquirer-prompt-confirm has been deprecated and renamed to prompt-confirm.

type: 'confirme',

Misspelling of confirm
After using prompt-confirm and correcting the typo it's working for me.

After using prompt-confirm and correcting the typo it's working for me.

same, I didn't notice the typo at first

Hi,
OMG I'm so sorry, for the confirme vs. confirm mistake!
As for the fact that enquirer-prompt-confirm has been deprecated and renamed to prompt-confirm, could you make it appear on the README page, like you did for enquirer-prompt-radio here it's plainly clear that the package is deprecated and renamed, but it's not for enquirer-prompt-confirm.
I have fixed my mistake and used the new package, my test case works ok now. I haven't tried chaining the calls, but I don't see any reason why it would not work.
Thanks again for your work and reactivity guys! And sorry for the typo :/
(Please consider editing the README page for the deprecated package!)


Quality of life

A few quality of life fixes picked up as I set up my development environment for the first time, and created my first python patch. Nothing too exciting here.

Thanks!


DRILL-3918: During expansion save the metadata for future use.

+1
This is basically the same fix I was planning.


Population must be a sequence.  For dicts or sets, use sorted(d).

Hello, I found an error after running your example: TicTacToe as follows:
TypeError                                 Traceback (most recent call last)
Cell In[3], line 2
      1 tree = UCT(game, allow_transpositions = True, training = True)
----> 2 tree.self_play(iterations = 500)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\mcts_simple\mcts\mcts.py:119, in MCTS.self_play(self, iterations)
    117 desc = "Training" if self.training is True else "Evaluating"
    118 for _ in tqdm(range(iterations), desc = desc):
--> 119     self.step()

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\mcts_simple\mcts\mcts.py:100, in MCTS.step(self)
     98 def step(self) -> None:
     99     if self.training is True:
--> 100         self.backpropagation(self.simulation(self.expansion(self.selection(self.root))))
    101     else:
    102         node = self.root

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\mcts_simple\mcts\uct.py:43, in UCT.expansion(self, path)
     42 def expansion(self, path: List[UCTNode]) -> List[UCTNode]:
---> 43     return super().expansion(path)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\mcts_simple\mcts\mcts.py:74, in MCTS.expansion(self, path)
     71 assert len(path[-1].children) > 0
     73 path[-1].is_expanded = True
---> 74 action = path[-1].choose_random_action()
     75 path.append(path[-1].children[action])
     76 self.copied_game.take_action(action)

File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\mcts_simple\mcts\mcts.py:43, in Node.choose_random_action(self)
     42 def choose_random_action(self) -> int:
---> 43     return random.sample(self.children.keys(), 1)[0]

File ~\AppData\Local\Programs\Python\Python311\Lib\random.py:439, in Random.sample(self, population, k, counts)
    415 # Sampling without replacement entails tracking either potential
    416 # selections (the pool) in a list or previous selections in a set.
    417
   (...)
    435 # too many calls to _randbelow(), making them slower and
    436 # causing them to eat more entropy than necessary.
    438 if not isinstance(population, _Sequence):
--> 439     raise TypeError("Population must be a sequence.  "
    440                     "For dicts or sets, use sorted(d).")
    441 n = len(population)
    442 if counts is not None:

TypeError: Population must be a sequence.  For dicts or sets, use sorted(d).

I'd appreciated if you have any proper workaround. Thank you.

Hi Vincent, it seems like you are using Python 3.11 which is currently not yet supported by mcts_simple, but you can try it out and open another issue if anything else goes wrong.
In this particular case, it seems like a version-only problem. It works for Python 3.7-3.10, but fails for Python 3.11. I noticed that doing a random sample on the dictionary keys in Python 3.10 will produce DeprecationWarning: Sampling from a set deprecated since Python 3.9 and will be removed in a subsequent version., but the code should still run nonetheless.
I would like to refer you to the following block of code. Replace line 43 with return random.sample(list(self.children), 1)[0] and it should work.
File ~\AppData\Local\Programs\Python\Python311\Lib\site-packages\mcts_simple\mcts\mcts.py:43, in Node.choose_random_action(self)
     42 def choose_random_action(self) -> int:
---> 43     return random.sample(self.children.keys(), 1)[0]

This will be updated soon, thanks for pointing this out. Do close this issue if the fix works. 🍡

Hi there, just added the relevant commit and updated the module on PyPI. Cheers!

Thank you!


Is there a way to use oracle without DBA privileges

First of all thanks for the Great Framework
what i see is DBMS_LOB access is a access in most is a concern from the DBA teams , Can you please suggest if i can use some other alternative .

This project is a fork of http://code.google.com/p/elmah, and this for has had Oracle compatibility removed. You may want to check with the original author.


IE compatibility

Prefer IE 10 instead of 9.

You're right, I was just going for the easiest fix. I don't think we even need this tag:
https://stackoverflow.com/questions/26346917/why-use-x-ua-compatible-ie-edge-anymore
And it came with the very first commit.


sleep on page without losing focus

Hello folks,
I have a code that navigate to the url and need to wait 5/10 seconds with keeping focus on to execute/validade some Javascript after...
The way I have found to sleep in a page without close the process is to use the :timer.sleep\1
but when I use :timer.sleep\1  the pages lose the Focus and invalidate by test case
note: The page has a counter, that count decreasing the time when I have focus on, when I lose focus on Page, I got a message like "Please keep this ad on focus..." like the image above.
The screenshot image when I navigate_to him:

The screenshot image after sleep for 15 seconds: ( :timer.sleep(15000))

Note when I tried to stay on Page in stand by using :timer.sleep\1 I just lose the focus
Thanks in advance and for awesome library

It seems unlikely that :timer.sleep\1 is causing your problem.  Hound commands send API calls to your webdriver of choice to interact with the page, and :timer.sleep\1 wouldn't do that.
Is there something specific that you're waiting to happen?  You might try repeatedly testing for that specific condition until it is true before proceeding.  You can send any Javascript you want with ScriptExecution.execute_script\1, and the return value of the script should be the return value of that function in Elixir.


cleanup(storage): no sessions for file uploads

This changes the Client::UploadFile() functions to upload without
using resumable sessions.  It is the first public API that uses the
session-less resumable uploads.
Part of the work for #8621

This change is

/FYI: @Jseph


Why is "pip install ." taking sooo long?

I think if you have output image files etc. in a subdirectory, they are included in the archive that pip makes so it can be slow...


Known issues regarding implementation.

The following are some problems with the code I noticed while I was trying to reproduce results of the original paper.

Public dataset and preprocessing methods should be updated for reproducing results in original paper.

The noise sampling method is different from the original code.
During reproducing the results, using the same noise sampling mechanism fails to fit the original dataset B * torch.uniform((S, Z))
https://github.com/birdx0810/timegan-pytorch/blob/7d0445540aed9f9e718c122d537535a172dd2f4f/models/utils.py#L114

Should be changed to something like the code below, and not torch.random((B, S, Z)) to follow a more Wiener Process.
Z_mb  = torch.zeros(B, S, Z)
for idx in batch_size:
    Z_mb[idx] = torch.random(S, Z)

The MSE losses are not respected to sequence length, this would make the model learn padding values when the sequence lengths are not of equal length. This is issue should be highlighted at all calculations of MSE, especially in recovery and supervisor forward pass. This should not be an issue if the public dataset is being used.
https://github.com/birdx0810/timegan-pytorch/blob/7d0445540aed9f9e718c122d537535a172dd2f4f/models/timegan.py#L414

G_loss is wrong in logging, accidental addition of torch.sqrt that is not in original code
https://github.com/birdx0810/timegan-pytorch/blob/7d0445540aed9f9e718c122d537535a172dd2f4f/models/utils.py#L120

Paddings should be added during inference stage.
https://github.com/birdx0810/timegan-pytorch/blob/7d0445540aed9f9e718c122d537535a172dd2f4f/models/utils.py#L271

Original code has a sigmoid activation function. Although Hide-and-Seek competition did not added this if I'm not mistaken, probably heuristics.
https://github.com/birdx0810/timegan-pytorch/blob/7d0445540aed9f9e718c122d537535a172dd2f4f/models/timegan.py#L150

Arguments if the loss should be instance or stepwise. To be experimented.
https://github.com/jsyoon0823/TimeGAN/issues/11#issuecomment-713323225

These issues have not been resolved as of now. Might take awhile to resolve as I have other things to settle.
Feel free to submit a PR.

Hello I would like to ask if the Time output in the data_preprocess.py file is treated as a category tag or as a time duration.

Hi @birdx0810, regarding point no. 5 above, was there a reason for excluding the sigmoid on the recovery network output?
Was it just to allow the network to learn real outputs instead of only (0, 1)? Or some other reason?

Hi @birdx0810, regarding point no. 5 above, was there a reason for excluding the sigmoid on the recovery network output?
Was it just to allow the network to learn real outputs instead of only (0, 1)? Or some other reason?

Sorry for the late reply, but this code base was derived from the NeurIPS 2020 hide-and-seek privacy challenge hosted by Van Der Schaar Lab, which is the team that proposed the TimeGAN model. I assume that it was just accidentally left out and I only realized it later. It should be added IMO, which is why I listed it in this issue.


roleAssignment still fails with principalType property, due to replication delay

Description
$group = New-AzADGroup -DisplayName $groupName -MailEnabled:$false -MailNickname $groupMailNickname -SecurityEnabled
New-AzRoleAssignment -Scope $scope -RoleDefinitionName $role -ObjectId $group.Id -ObjectType Group

Using the PowerShell cmdlets New-AzADGroup to create a new security group, and New-AzRoleAssignment to immediately try and assign a role to this group, fails with the exception message:
Exception
Code           : PrincipalNotFound
Message        : Principal [guid] does not exist in the directory [guid]. Check that you have the correct principal ID. If you are creating this principal and then immediately assigning a role, this error might be related to a replication delay. In this case, set the role assignment principalType property to a value, such as ServicePrincipal, User, or Group.  See https://aka.ms/docs-principaltype
Target         :
Details        :
AdditionalInfo :

Documentation
The documentation states similar to the error, to set the principalType property to resolve the error.

If you create a new service principal and immediately try to assign a role to that service principal, that role assignment can fail in some cases. For example, if you create a new managed identity and then try to assign a role to that service principal, the role assignment might fail. The reason for this failure is likely a replication delay. The service principal is created in one region; however, the role assignment might occur in a different region that hasn't replicated the service principal yet.
To address this scenario, use the Role Assignments - Create REST API and set the principalType property to ServicePrincipal. You must also set the apiVersion to 2018-09-01-preview or later. 2022-04-01 is the first stable version.

HTTP request captured from exception
Method     : PUT
RequestUri : https://management.azure.com//subscriptions/[guid]/resourceGroups/[resourceGroupName]/providers/Microsoft.Authorization/roleAssignments/34a80422-5d12-4062-8d48-b40454a7cda3?api-version=2020-08-01-preview
Properties : {}
Content    : {
               "properties": {
                 "roleDefinitionId": "/subscriptions/[guid]/resourceGroups/[resourceGroupName]/providers/Microsoft.Authorization/roleDefinitions/b24988ac-6180-42a0-ab88-20f7382dd24c",
                 "principalId": "[guid]",
                 "principalType": "Group"
               }
             }
Headers    : {[x-ms-client-request-id, System.String[]], [accept-language, System.String[]], [Authorization, System.String[]], [User-Agent, System.String[]]...}

The request shows that the principalType property has been set, and the apiVersion is 2018-09-01-preview or later
This issue is similar to #63173, although that issue is referring to a principalType of ServicePrincipal instead of Group

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 1e974af9-501a-8eac-b9c1-d3252120abbb
Version Independent ID: 8f659349-3ae6-1c39-faf4-49b70cb0db02
Content: Assign Azure roles using the REST API - Azure RBAC
Content Source: articles/role-based-access-control/role-assignments-rest.md
Service: role-based-access-control
GitHub Login: @rolyon
Microsoft Alias: rolyon

@scottwtang
Thanks for your feedback! We will investigate and update as appropriate.

@rolyon
Can you please check and add your comments on this doc update request as applicable.

Hi @scottwtang
Thanks for the detail about the error you are encountering. We've sent a message to the team to see if we can find out more about why this error occurs.
thanks

Hi @scottwtang
Thanks for the detail about the error you are encountering. We've sent a message to the team to see if we can find out more about why this error occurs.
thanks

Hi @scottwtang
When creating a new group, it looks like you'll need to wait a few minutes before you can assign a role. We've made an update to the docs.
https://learn.microsoft.com/en-us/azure/role-based-access-control/troubleshooting?tabs=bicep#symptom---assigning-a-role-to-a-new-principal-sometimes-fails
Hopefully this helps. Let us know if you have any additional feedback.
thanks

#please-close

Still having this issue using bicep. Any resolution?


Example List reference to itself wrong?

Section Number
https://profiles.ihe.net/ITI/MHD/List-593cd04e-b696-45c1-bc32-39e55a340a45.json.html
Issue
entry.item.reference points to the example list instance itself (593cd04e-b696-45c1-bc32-39e55a340a46), should this not be to a DocumentReference?
Proposed Change *
Fix example
Priority:

Low: Typo or other minor classification that an editor can manage. Requires no group discussion.

I think this is a Folder (0a45) that points at a DocumentReference (0a46)
see https://github.com/IHE/ITI.MHD/blob/master/input/fsh/ex-ProvideBundles.fsh
No question there is room for better clarity.


#1051: migrated all usages of models.MetadataUser & models.User classes to g…

Describe your changes :
migrated all usages of models.MetadataUser & models.User classes to generated.schema.teams.user.User classes instead. I This was in order to use only the native pydantic generated models.

Type of change :

[] Bug fix
[x] Improvement
[] New feature
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] Documentation

Frontend Preview (Screenshots) :
For frontend related change, please link screenshots of your changes preview! Optional for backend related changes.

Checklist:

[x] I have read the CONTRIBUTING document.
[x] I have performed a self-review of my own.
[x] I have tagged my reviewers below.
[x] I have commented on my code, particularly in hard-to-understand areas.
[x] My changes generate no new warnings.
[ ] I have added tests that prove my fix is effective or that my feature works.
[x] All new and existing tests passed.

@amiorin @pmbrull
Reviewers

Thanks @omriAl Looks good. +1.

Not all uses of the old User class have been replaced with the generated one.

fix here:
https://github.com/open-metadata/OpenMetadata/pull/1492


Fix broken image url

Open issue of ionic repo:
https://github.com/driftyco/ionic/issues/4175
Close #273

thanks!


Open source models

Try to sub out GPT-4V for an open source model like CogVLM, Fuyu-8B, Qwen-VL, or LLaVA.

especially for the more delirious prompts


Streaming still writes temporary files to disk

In issue #14 you stated...

All you would need to do is download the file into a Stream object (or byte[]) in memory and then pass it to the SendAndScanFile(Stream sourceStream) or SendAndScanFile(byte[] fileData) methods of ClamClient. These methods never write the file to disk and stream them from memory.

However, as far as I can tell, ClamAV is still writing file contents to the temp folder even when SendAndScanFileAsync, which I can see uses INSTREAM when calling clamd. I'm not sure that this is a limitation of nClam but I'm wondering if there's some configuration setting to ensure that nothing is ever written to disk. Any ideas/suggestions?

To clarify, nClam will never persist the file to disk and will stream the data directly to ClamD.
It seems you are saying that ClamD may temporarily store the file on disk when scanning it.  I'm not sure if it does this, but if that is wrong it would be great to clarify this for users - please reopen this issue and reference the ClamD docs so we can better document those calls.


Update Badges in README.md

This PR solves this issue by always taking the latest release tag from github 👍


TypeError: AgentLLM.update_output_list() missing 1 required positional argument: 'output'

Running oobabooga local with vicuna as the model with the latest pull. I can create an agent, set it's objective and get a response back but at some point this error pops up and the agent just loops task over and over.
Exception in thread Thread-6 (run_task):
Traceback (most recent call last):
  File "C:\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 282, in run_task
    task = self.execute_next_task()
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 257, in execute_next_task
    self.response = self.execution_agent(self.primary_objective, this_task_name, this_task_id)
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 229, in execution_agent
    self.update_output_list(f"Execution agent response:\n\n{self.response}")
TypeError: AgentLLM.update_output_list() missing 1 required positional argument: 'output'

I believe solved by #59

This particular error is gone now. There's a new one in its place.
Exception in thread Thread-6 (run_task):
Traceback (most recent call last):
  File "C:\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 282, in run_task
    task = self.execute_next_task()
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 257, in execute_next_task
    self.response = self.execution_agent(self.primary_objective, this_task_name, this_task_id)
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 230, in execution_agent
    print(self.output_list)
AttributeError: 'AgentLLM' object has no attribute 'output_list'

This particular error is gone now. There's a new one in its place.
Exception in thread Thread-6 (run_task):
Traceback (most recent call last):
  File "C:\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 282, in run_task
    task = self.execute_next_task()
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 257, in execute_next_task
    self.response = self.execution_agent(self.primary_objective, this_task_name, this_task_id)
  File "D:\LocalGPT\Agent-LLM\AgentLLM.py", line 230, in execution_agent
    print(self.output_list)
AttributeError: 'AgentLLM' object has no attribute 'output_list'

Just pushed an update to fix this.

Running a test now.


[APP] Resize support

Builds upon #1120
Fixes #1104

Note: Dx11 resize doesn't work properly yet, but it's no fatal. See  #1122
@homu r+

:pushpin: Commit 90e632f has been approved by kvark

:zap: Test exempted - status


Added: Student Dashboard Screen

Issue Number
fixes #135

Describe the changes you've made
Created Student Dashboard Screen and some of the components are left to add like

[x] Assignment Submission Section
[ ] Job page

I'll create a separate  PR for the respective sections that will resolve this issue!
Describe if there is any unusual behavior (Any Warning) of your code(Write NA if there isn't)

Additional context (OPTIONAL)

Test plan (OPTIONAL)
A good test plan should give instructions that someone else can easily follow.

Checklist

[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] All new and existing tests passed.
[x] The title of my pull request is a short description of the requested changes.

@Abhishek-kumar09 ping!

@adarsh-technocrat LGTM 👍
great work!!

Follow camelcasing while naming variables or classes etc.

Thanks, @Abhishek-kumar09  I'll keep in mind :)

Sorry, @Abhishek-kumar09 hitted accidentally close with the comment option!

@Abhishek-kumar09  ping!

There are merge conflicts.
You can resolve these in browser itself.
This extension also helps: https://addons.mozilla.org/en-US/firefox/addon/refined-github-/

There are merge conflicts.
You can resolve these in browser itself.
This extension also helps: https://addons.mozilla.org/en-US/firefox/addon/refined-github-/

Thanks @Abhishek-kumar09  I'll do the changes :)

Also, create a new PR to update PR template which asks for giving links to the deploy preview of the route/page to be reviewed

Also, create a new PR to update PR template which asks for giving links to the deploy preview of the route/page to be reviewed

Sure @Abhishek-kumar09  I'll create PR for that :)

https://user-images.githubusercontent.com/47661086/114194469-28cde800-996d-11eb-8f31-40461e9fa240.mp4

Good Work!


Use Theme brightness instead of FadeTheme

Instead of using a custom enum to define dark/light theme, let's instead use Theme.of(context).brightness to decide colors, since it's more aligned with official flutter widgets. The colors from Theme will be used as fallback colors in case baseColor and highlightColor aren't defined.
This shouldn't affect how the widget looks.

the FadeTheme is used for dev to choose pre defined colors, so i'll keep it.


Update solution rules yamls and adding 1 playbook

Fixes #
Proposed Changes

@aprakash13 pushed fixed including techniques back