Deserializing models can take a long time on some Android devices

These models are taking a long time to load on a Google Pixel 6 phone, even though they are embedded into the build with MLModelDataEmbed:

https://hub.natml.ai/@natml/blazepose-detector: ~6 seconds
https://hub.natml.ai/@natml/blazepose-landmark: ~14 seconds

We have found by profiling that almost all of this time is spent in NatML.CreateModel.
On a Samsung Galazy S21 Ultra the same models take only ~1.5 seconds to load in total.
Is there a format conversion or any expensive computation happening here? If so, is it possible to do this conversion ahead of time (such as when the app starts) without having to keep the model in memory for the entire lifetime of the app?

@joseph-o3h , what happens when you add the following line?
var modelData = ...
modelData.computeTarget = MLModelData.ComputeTarget.CPUOnly; // <-- add this
var model = new MLEdgeModel(modelData);

I suspect the time is being spent creating an NNAPI representation of the model for execution.

@joseph-o3h , what happens when you add the following line?
var modelData = ...
modelData.computeTarget = MLModelData.ComputeTarget.CPUOnly; // <-- add this
var model = new MLEdgeModel(modelData);

I suspect the time is being spent creating an NNAPI representation of the model for execution.

I cannot use computeTarget right now as it was added in 1.0.18, but we are still on 1.0.13 because of https://github.com/natmlx/NatML/issues/35

@joseph-o3h working on #35 . We're batching a few things into the next update. Regarding this issue, we've changed the API to make model creation asynchronous:
// Fetch the model data
var modelData = await MLModelData.FromHub(tag);
// Create the model
var model = await MLEdgeModel.Create(modelData); // <-- this is offloaded to a native background worker

But even with the asynchronous model creation it will take the same amount of time won't it? It is just not going to block the main thread.

But even with the asynchronous model creation it will take the same amount of time won't it? It is just not going to block the main thread.

That's correct, though the time taken should be on the order of a few frames.

That's correct, though the time taken should be on the order of a few frames.

Is this when you are running the model on the CPU? That might improve the load time but could also impact run time performance.
NNAPI supports caching of compiled models (https://developer.android.com/ndk/reference/group/neural-networks#aneuralnetworkscompilation_setcaching), is it possible to use it in NatML (if it is not being used already)?

Hey @joseph-o3h Happy New Year! I've got inline responses below:

Is this when inference is run on the CPU? That might improve the model load/compile time but could also impact run time performance.

That's correct!

NNAPI supports caching of compiled models (https://developer.android.com/ndk/reference/group/neural-networks#aneuralnetworkscompilation_setcaching), is it possible to use it in NatML (if it is not being used already)?

NatML doesn't use NNAPI caching unfortunately. Adding support for IR caching is on the mid-to-longer term roadmap.

We've had an engineering slowdown over the holidays, but we're picking back up now. ETA on the update with async model creation should be sometime next week.

Okay minor follow up on the caching question: we're likely gonna add support for this in the near-term, but to iOS, macOS, and Windows first.

Hey @joseph-o3h we've updated model creation to be async in the NatML 1.1 update. For device-specific delays in creating the model, the culprit is likely building the NNAPI representation, so you can either keep the model on the CPU (won't  advise this); or you can hide the delay since the process is now async. I'm closing this issue; feel free to reopen another issue if you run into something similar.


[VSCODE FLEX JS] Error launching SWF debug session.

Hi,
I am trying to create a sample Flex JS project by following the tutorial given on git hub
https://github.com/BowlerHatLLC/vscode-nextgenas/wiki/Create-a-new-ActionScript-project-in-Visual-Studio-Code-that-targets-FlexJS
Project compiles successfully. However, when i try to debug the project it gives error
"Error launching SWF debug session. Runtime not found for program: c:\VisualCode/Main.swf"
Thanks in advance.
Regards

I noticed that you're trying to work on a FlexJS project, but you're launching a SWF file. That's a perfectly valid workflow, since FlexJS can target both SWF and JS, but I'm curious if that's your intention. Do you actually want to create a SWF? Or do you want a pure JS/HTML app?
If you do want to create a SWF and launch it directly (not embedded in HTML), you'll need to install the Flash Player projector content debugger from Adobe Flash Player Debug Downloads and make sure that it's associated with .swf files on your system. For more information, see this tutorial: Debug a SWF in the standalone Adobe Flash Player with Visual Studio Code
If you're actually embedding the SWF file in HTML, then you'll need to launch the HTML file instead.
If you prefer to target JS/HTML instead of Flash Player, then you don't want to launch a SWF at all. I've written tutorials about how to debug in Chrome and Firefox.

Josh,
Thank you very much for your reply.
I am trying to configure JS/HTML app.
After posting my question, i watched your video on youtube. Which was uploaded on "The linux foundation". There was step by step procedure on how to setup visual code environment for FLex JS (Thanks to you). I followed the same procedure. However after setting the  workspace preferences i.e "nextgenas.sdk.editor" value to path where flex JS is installed. It gives error which states
"Failed to load project configuration option. Error checking disabled, except for single syntax problems(1,1)"

Also in video it is shown that to compile the project View -> Command palette -> Tasks : Configure Task Runner ( ActionScript - asconfig.json) (DEPRECATED)

Please note that (DEPRECATED) word is written in VS Code with configure task runner. I am using VS Code 1.14.0. After selecting this option another screen appears which says select a task runner. There are four options, by selecting any of them doesn't produce the "tasks.json" file as shown in video.

Regards

Unfortunately, this video is presents some out-of-date information. However, it may still be somewhat helpful.

However after setting the workspace preferences i.e "nextgenas.sdk.editor" value to path where flex JS is installed. It gives error which states
"Failed to load project configuration option. Error checking disabled, except for single syntax problems(1,1)"

It's worth mentioning that most people shouldn't need to use the nextgenas.sdk.editor setting anymore, unless they're using "nightly" builds of FlexJS with new compiler changes. Generally, you should only need to choose an SDK as described here.
I noticed that the filename of your MXML class has a typo (MyFirstFlexJSProect instead of MyFirstFlexJSProject). I suspect that this is why you're getting that error. Can you verify that the file name matches what's specified in your asconfig.json file?

Please note that (DEPRECATED) word is written in VS Code with configure task runner. I am using VS Code 1.14.0. After selecting this option another screen appears which says select a task runner. There are four options, by selecting any of them doesn't produce the "tasks.json" file as shown in video.

That's odd. That list should contain additional options for ActionScript. Regardless, it's not the recommended way to build your project anymore. Please take a look at Build an ActionScript project in Visual Studio Code for the latest instructions.

I also just noticed another mistake in your screenshot. It appears that the src directory and asconfig.json are inside the .vscode directory. These should be in the root instead.

If the asconfig.json file is in the wrong location, that would definitely explain why you saw the wrong items with the "Tasks : Configure Task Runner ( ActionScript - asconfig.json) (DEPRECATED)" command. In the next version, I think I'll add a warning if the asconfig.json file is not found so that this is more obvious.

Josh,
Thank you for your reply.
Yes, the first error was typo. Secondly, asconfig and src were not where they meant to be. Although you mentioned these in video(Apologies).
I made all the amendments you suggested and build the project according to the edited tutorial. And it worked like charm.
Thanks for all your efforts and time.
Best Regards

Happy to help!


Negative dimension size ValueError

Hi Felix, thanks for the code you have provided here!
I just clone everything and am attempting to run things out of box before doing my own finetuning .. I'm running into this issue though, and am wondering if you have any insights.
I simply run 'python vgg16.py' and get this output:
Kristas-MacBook-Pro-2:cnn_finetune Krista$ python vgg19.py
Using TensorFlow backend.
2017-11-10 12:30:18.227736: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
vgg19.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(64, (3, 3), activation="relu")
model.add(Convolution2D(64, 3, 3, activation='relu'))
vgg19.py:31: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(64, (3, 3), activation="relu")
model.add(Convolution2D(64, 3, 3, activation='relu'))
vgg19.py:35: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(128, (3, 3), activation="relu")
model.add(Convolution2D(128, 3, 3, activation='relu'))
vgg19.py:37: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(128, (3, 3), activation="relu")
model.add(Convolution2D(128, 3, 3, activation='relu'))
Traceback (most recent call last):
File "vgg19.py", line 108, in
model = vgg19_model(img_rows, img_cols, channel, num_classes)
File "vgg19.py", line 38, in vgg19_model
model.add(MaxPooling2D((2,2), strides=(2,2)))
File "/usr/local/lib/python2.7/site-packages/keras/models.py", line 475, in add
output_tensor = layer(self.outputs[0])
File "/usr/local/lib/python2.7/site-packages/keras/engine/topology.py", line 603, in call
output = self.call(inputs, **kwargs)
File "/usr/local/lib/python2.7/site-packages/keras/layers/pooling.py", line 154, in call
data_format=self.data_format)
File "/usr/local/lib/python2.7/site-packages/keras/layers/pooling.py", line 217, in _pooling_function
pool_mode='max')
File "/usr/local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 3456, in pool2d
data_format=tf_data_format)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/nn_ops.py", line 1958, in max_pool
name=name)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 2806, in _max_pool
data_format=data_format, name=name)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2958, in create_op
set_shapes_for_outputs(ret)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2209, in set_shapes_for_outputs
shapes = shape_func(op)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2159, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn
require_shape_fn)
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Negative dimension size caused by subtracting 2 from 1 for 'max_pooling2d_2/MaxPool' (op: 'MaxPool') with input shapes: [?,1,112,128].
I'm using python 2..7.14, keras 2.0.9, and tensorflow 1.4.0
Thanks in advance!

I am also getting the same error!!

@vaghelan I never actually got this error fixed, but I found that some of the other NN architectures (like the resnet_50 and inception_v4) worked for me instead! I'd still love to get the VGG going though, so let me know if you come up with anything that helps...

error for resnet_50 also
resnet_50.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(256, (1, 1), name="res4f_branch2a")
x = Convolution2D(nb_filter1, 1, 1, name=conv_name_base + '2a')(input_tensor)
resnet_50.py:34: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(256, (3, 3), padding="same", name="res4f_branch2b")
border_mode='same', name=conv_name_base + '2b')(x)
resnet_50.py:38: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(1024, (1, 1), name="res4f_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
resnet_50.py:63: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (1, 1), strides=(2, 2), name="res5a_branch2a")
name=conv_name_base + '2a')(input_tensor)
resnet_50.py:68: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (3, 3), padding="same", name="res5a_branch2b")
name=conv_name_base + '2b')(x)
resnet_50.py:72: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), name="res5a_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
resnet_50.py:76: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), strides=(2, 2), name="res5a_branch1")
name=conv_name_base + '1')(input_tensor)
resnet_50.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (1, 1), name="res5b_branch2a")
x = Convolution2D(nb_filter1, 1, 1, name=conv_name_base + '2a')(input_tensor)
resnet_50.py:34: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (3, 3), padding="same", name="res5b_branch2b")
border_mode='same', name=conv_name_base + '2b')(x)
resnet_50.py:38: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), name="res5b_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
resnet_50.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (1, 1), name="res5c_branch2a")
x = Convolution2D(nb_filter1, 1, 1, name=conv_name_base + '2a')(input_tensor)
resnet_50.py:34: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (3, 3), padding="same", name="res5c_branch2b")
border_mode='same', name=conv_name_base + '2b')(x)
resnet_50.py:38: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), name="res5c_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
Traceback (most recent call last):
File "resnet_50.py", line 182, in
model = resnet50_model(img_rows, img_cols, channel, num_classes)
File "resnet_50.py", line 150, in resnet50_model
model.load_weights(weights_path)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2532, in load_weights
f = h5py.File(filepath, mode='r')
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 271, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 101, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2798)
File "h5py/h5f.pyx", line 78, in h5py.h5f.open (/tmp/pip-nCYoKW-build/h5py/h5f.c:2117)
IOError: Unable to open file (Unable to open file: name = 'imagenet_models/resnet50_weights_tf_dim_ordering_tf_kernels.h5', errno = 2, error mes
sage = 'no such file or directory', flags = 0, o_flags = 0)

@vaghelan I know the fix to that problem! You just need to make sure you download the resnet50_weights_tf_dim_ordering_tf_kernels.h5 file () and place it in the imagenet_models directory. The download can be found here >> https://github.com/fchollet/deep-learning-models/releases.

New error...
resnet_50.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(256, (1, 1), name="res4f_branch2a")
x = Convolution2D(nb_filter1, 1, 1, name=conv_name_base + '2a')(input_tensor)
resnet_50.py:34: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(256, (3, 3), padding="same", name="res4f_branch2b")
border_mode='same', name=conv_name_base + '2b')(x)
resnet_50.py:38: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(1024, (1, 1), name="res4f_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
resnet_50.py:63: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (1, 1), strides=(2, 2), name="res5a_branch2a")
name=conv_name_base + '2a')(input_tensor)
resnet_50.py:68: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (3, 3), padding="same", name="res5a_branch2b")
name=conv_name_base + '2b')(x)
resnet_50.py:72: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), name="res5a_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
resnet_50.py:76: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), strides=(2, 2), name="res5a_branch1")
name=conv_name_base + '1')(input_tensor)
resnet_50.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (1, 1), name="res5b_branch2a")
x = Convolution2D(nb_filter1, 1, 1, name=conv_name_base + '2a')(input_tensor)
resnet_50.py:34: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (3, 3), padding="same", name="res5b_branch2b")
border_mode='same', name=conv_name_base + '2b')(x)
resnet_50.py:38: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), name="res5b_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
resnet_50.py:29: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (1, 1), name="res5c_branch2a")
x = Convolution2D(nb_filter1, 1, 1, name=conv_name_base + '2a')(input_tensor)
resnet_50.py:34: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(512, (3, 3), padding="same", name="res5c_branch2b")
border_mode='same', name=conv_name_base + '2b')(x)
resnet_50.py:38: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(2048, (1, 1), name="res5c_branch2c")
x = Convolution2D(nb_filter3, 1, 1, name=conv_name_base + '2c')(x)
Traceback (most recent call last):
File "resnet_50.py", line 182, in
model = resnet50_model(img_rows, img_cols, channel, num_classes)
File "resnet_50.py", line 150, in resnet50_model
model.load_weights(weights_path)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2532, in load_weights
f = h5py.File(filepath, mode='r')
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 271, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/usr/local/lib/python2.7/dist-packages/h5py/_hl/files.py", line 101, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2840)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-nCYoKW-build/h5py/_objects.c:2798)
File "h5py/h5f.pyx", line 78, in h5py.h5f.open (/tmp/pip-nCYoKW-build/h5py/h5f.c:2117)
IOError: Unable to open file (Unable to open file: name = 'imagenet_models/resnet50_weights_tf_dim_ordering_tf_kernels.h5', errno = 2, error mes
sage = 'no such file or directory', flags = 0, o_flags = 0)

New error :
Traceback (most recent call last):
File "resnet_50.py", line 182, in
model = resnet50_model(img_rows, img_cols, channel, num_classes)
File "resnet_50.py", line 150, in resnet50_model
model.load_weights(weights_path)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2538, in load_weights
load_weights_from_hdf5_group(f, self.layers)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 2970, in load_weights_from_hdf5_group
K.batch_set_value(weight_value_tuples)
File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 2148, in batch_set_value
assign_op = x.assign(assign_placeholder)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 512, in assign
return state_ops.assign(self._variable, value, use_locking=use_locking)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/state_ops.py", line 270, in assign
validate_shape=validate_shape)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2338, in create_op
set_shapes_for_outputs(ret)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1719, in set_shapes_for_outputs
shapes = shape_func(op)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1669, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
debug_python_shape_fn, require_shape_fn)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Dimension 2 in both shapes must be equal, but are 128 and 256 for 'Assign_78' (op: 'Assign') with input shapes: [1,1,128,512], [1,1,
256,512].

@vaghelan how many channels are you using? And is your input data still just the cifar data that was given by default?

I just typed : python resnet_50.py

Not entirely sure on this issue, but given that the error is in load weights, my best guess is that the weight file is incorrect for some reason and you need to search for another one (the file is online in lots of different locations)

I think code is old way of doing fine tuning. With new version of keras we have to do from keras.applications
https://keras.io/applications/
Look at fine tuning section.
They have given example for Inceptionv3. They knock off top layer and add new layer with less classes.

Same problem with me.
I am using resnet-152 architecture and training input of size 224x224 and 331 images with tensor-flow backend but keep getting this error:
ValueError: Error when checking input: expected data to have shape (None, 224, 224, 3) but got array with shape (331, 3, 224, 224)
I tried setting image data format to channel_first and channel_last but its not working either way
Kindly help me out

I am also getting the same error
I'm using python 3.7, keras 2.2.4, and tensorflow 1.8.0
I modification code
model.add(ZeroPadding2D((1, 1), input_shape=(channel,img_rows, img_cols))) == >> model.add(ZeroPadding2D((1, 1), input_shape=(img_rows, img_cols, channel)))
channel last
and Convolution2D modification
model.add(Convolution2D(128, kernel_size=3, activation='relu',padding='same'))
The code works


Issues running PyAF

Hi @antoinecarme

Like I mentioned earlier I have the same problem as the one on this thread. I don't know if the kernel hangs or it is taking too much time to train the data. My data is just a pandas dataframe of two columns with a date time column of 365 days and respective sales column for those dates. I have arranged everything in the exact format as you mentioned in the PyAF documentation and sample code.
I have attached the snapshot..

After installing PyAF, I got this error while trying to load some packages - 'tuple object has no attribute type'.
I got it resolved after finding that numpy, pandas and sklearn packages got installed again with different versions during the installation of pyAF and other mentioned packages. Also other than those three many other packages didn't load throwing the same error. I don't know which one is doing this.

Is there a clear description of what auto forecast engine is doing in the background or is it a black box? How the training is done and what's the metric it takes for validation. Also it would be great if you could provide an article on how auto forecast is doing differently or if better than other popular forecasting techniques like ARIMA, prophet etc.
Thank you.

Originally posted by @anuliketat in https://github.com/antoinecarme/pyaf/issues/39#issuecomment-456598548

Closing issue after no response for 30 days. Not blocking.
Please repoen if needed.


Fixes Broken Pages

Fixes Broken Pages due to #1094
Describe the changes you have made in this PR -
I have changed some alt= to alt:
Please let me know if anymore changes are required here.

Coverage remained the same at 84.266% when pulling 830a430f7564205f234e52a5058243bd7a7d2b1d on Shivansh2407:patch-31 into 95c7202f60c7a03bcf02d396e9bae0d43f00dbdb on CircuitVerse:master.

Fixed Now @tachyons .


Enhancement: Add a deployment list component

Adds a new deployment list component to match the flow list component.  This would replace the DeploymentsTable component and be shown on the deployments page.

I know we possibly have more great work coming for the deployments page but this felt like a quick upgrade to match flows.
@brandonreid - for design feedback
@WillRaphaelson and @billpalombi - I already showed Will but flagging for info

@zhen0 This is 🔥!
I think this is definitely an improvement as is, but here are a few design decisions for your consideration:

I believe we generally place status indicators after the text (in the case of the deployment name)
In this mock, I’ve put the flow name below the deployment so there’s some proximity and tighter association there.
This mock has a “Work Pool” column that also shows the work queue, this is from a previous design exploration, so feel free to include if it seems valuable, or totally dismiss.
The “Applied by” name and the date have been combined into one “Created” column.
The columns have been sorted a bit.

This makes me wonder if the Flows page is currently showing tags or not.. We probably should be. 🤔

I agree with nearly all of @brandonreid's proposals. That said, I don't want the perfect to be the enemy of the good. @zhen0, I think the only blocking changes are:

The status indicator does need to be to the right of the deployment name as it is elsewhere
We should remove the applied by column entirely. I know that came from user feedback, but I think we over-indexed on it. That information does not belong on this page.

With Bill's updates:

I'm going to merge this as is and we can continue discussion about icon placement elsewhere if we think it important.

I agree both with moving the status indicator and with treating that as a seperate PR. Thanks @zhen0 and @aaazzam!


Code Release

When do you intend to release the code?

Thanks for your interest! We are currently overwhelmed by CVPR submissions. We plan to release the code around mid-November.

Great work. Looking forward to the code as well.
In the meantime, good luck for the CVPR submission :)

Any update, please?

Thanks for your interest. We are now doing the final cleaning, code will be released as soon as possible after the holidays. Thanks :)

The codes are now released! Sorry for the delay. Please let me know if you encounter any problems when running the codes :)


PK adjustment for IdentityUserRole (AspNetUserRoles)

Looks like MultiTenantIdentityDbContext doesn't add TenantId to PK_AspNetUserRoles. Is it intended, and if so, why?
The workaround should be as trivial as calling .AdjustKeys(builder) for the IdentityUserRole but it's internal so its code has to be copied to OnModelCreating of the MultiTenantIdentityDbContext inheritor, which is not ideal.

Looks like I've got the answer:
System.ArgumentException: Entity type 'IdentityUserRole<Guid>' is defined with a 3-part composite key, but 2 values were passed to the 'Find' method.
   at Microsoft.EntityFrameworkCore.Internal.EntityFinder`1.ValidateKeyPropertiesAndExtractCancellationToken(Object[] keyValues, Boolean async, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Internal.EntityFinder`1.FindAsync(Object[] keyValues, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Internal.InternalDbSet`1.FindAsync(Object[] keyValues, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.UserStore`9.FindUserRoleAsync(TKey userId, TKey roleId, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Identity.EntityFrameworkCore.UserStore`9.IsInRoleAsync(TUser user, String normalizedRoleName, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Identity.UserManager`1.AddToRoleAsync(TUser user, String role)

So, effectively, I need to reimplement UserStore and pass tenantId in addition to userId & roleId?

Yeah you are seeing some of the limits of trying to tack on the tenantId column to these data models. Unfortunately there are a few methods in UserManager and UserStore and their role equivalents that use Find on DbSet which is problematic. Implementing your own classes which adjust just those cases is feasible. I’m planning to overhaul Identity support to address these but it’s still a ways off.

I got it to work. Not much additional code but took some time to figure out why it's not working out of the box. Would you be interested in a PR with the missing pieces?


Support find all references and rename with notebooks

Notebooks can pull in python imports into their cells.
Like so:
test.py:
def foo():
    print('foo')

test.ipynb: cell
import test
foo()

When writing a language server, it would be nice to support renaming and find all references such that these renames get applied to python cells in a notebook too.
In order for a language server to implement this it needs to predict the URI of cells in a notebook though. Otherwise the rename responses will have invalid URIs.
So this request is really to expose an API to give back the URI for cells of a notebook.

@rchiodo I misread, sorry about that and now move back to VS Code.

The URI format of cells (I think?) is opaque to everyone except for VS code

I think this is intentional when designing the fragment.

I think this is intentional when designing the fragment.

I think this being the case makes it impossible for say Visual Studio to implement something like this. We could just copy the code that VS code has for computing cell fragments, but then Visual Studio would also have to have that same code.
I feel like the cell uri should be spec'd somewhere (like the LSP), then it would follow a consistent pattern for all parties involved.

So this request is really to expose an API to give back the URI for cells of a notebook.

This exists via notebookCell.document.uri - no reason to specify the exact format of the uri. A working sample for rename and reference search can be found here: https://github.com/microsoft/vscode-github-issue-notebooks/blob/fe51aafec419dfa5ba99d2ed4795fcf2ea38e759/src/extension/languageProvider.ts#L130

it's the underlying issue with anybody being able to read notebooks when they're closed.

Given that you can read closed notebooks (aka plain text files) why do you need cell uris to begin with? You can just emit a "normal" text edit, right?

You can just emit a "normal" text edit, right?

Unless I'm missing something, the URI for the text edit would not be correct. Wouldn't it need the URI for the cell?

That example project you had looks like it's working with open notebooks. Not closed or just plain files on disk.

I don't think we want to handle open and close notebook or any other files differently. open and closed file is just state of UI not the file itself. I believe asking LS to handle those differently make all LS to reinvent the wheel. Either vscode or LSP should let LS to handle those 2 cases same. If we report location of symbol in cell using cell uri, it should be same for both open and closed files/notebook. if we get change notification of cells based on cell's position, it should be same for open/close notebook/files. if we report text edits on cells using cell's coordinates, it should be same for open/close notebook/files.
Otherwise, LSP is no longer client independent. and has to know all implementation details on how client handle open notebooks/files to mimic its behavior for closed files.
we opened similar issue on LSP side as well - https://github.com/microsoft/vscode-languageserver-node/issues/1209

In my opinion, asking LS to handle closed notebook file themselves is asking it to re-implement its own version of notebook serializer (https://code.visualstudio.com/api/references/vscode-api#NotebookSerializer). otherwise, open and close notebook's support will be different and probably different per clients as well.

In my opinion, asking LS to handle closed notebook file themselves is asking it to re-implement its own version of notebook serializer

I do agree. While this could work for jupyter it will most certainly fail for any other notebook type. However, in return this means a LS that wants to integrate with notebooks needs to open them (as model, not UI objects). I think that's fair, VS Code will auto-clean, and that also solves the problem of needing to synthesise cell uris.
@rchiodo Are you OK with this and happy to close without action?

@rchiodo Are you OK with this and happy to close without action?

I'm not sure what 'this' means?
Are you proposing LSP will handle this? I mean we're working with Dirk on adding a file API to LSP. Wouldn't be too hard to add a notebook model on top of that.
Or will VS code provide an API to read notebooks for extensions? Or maybe you mean something else entirely.

I mean close this issue without further action.
I am proposing that extensions use openNotebookDocument and the LSP equivalent. That ensures you have the right model object at hands with which you can work, without needing the ability to deserialise notebooks

Sorry I'm still not sure I understand your proposal.
As a specific example, what would Pylance do to handle jupyter notebooks that aren't open?
Or are you saying this issue it too specific and we should open a new one that is more generic to being able to read notebook models from closed files?

what would Pylance do to handle jupyter notebooks that aren't open

It would open them

@jrieken I am not quite sure. so we just silently open all notebook files? is that what you are suggesting? and just ignore if user actually open or close the notebook somehow?

Yeah, because there is no way to know how to deserialise a notebook it's your only option. Internally notebooks are ref-counted and managed by VS Code, meaning you call open, listen to to open/close events, but you cannot close them.

suggesting we open and process notebook and close them whenever there is request such as find all references, rename, workspace symbols and etc?

I believe that's up the internals of your language service. You could digest them once and keep the information around or always re-read files.

if we need to implement any custom code in client (vscode) side, won't it be better if we just use notebook serializer (that are registered to vscode) directly from vscode side instead of messing with notebook open or close ? (assuming we can do that and using serializer won't affect any states unlike open/close notebook)?

There is no real API for using serializers. There is two API commands (vscode.executeNotebookToData and vscode.executeDataToNotebook) but they work, just like serializers, on the level of vscode.NotebookData and NotebookCellData which have no uris rendering them useless for language servers.


[1.2] FabricDescriptorStruct type error in OperationalCredentialsCluster on Android Platform

Reproduction steps

invoke command readFabricsAttribute of OperationalCredentialsCluster
get result from android sdk:
result: Endpoint 0: { Cluster 62: { Attribute 1: {"1:ARRAY-STRUCT":[{"1:BYTES":"BDx99TpkVj+pForMbMO+Tm1M7MiSiJvXND+rxiJ2cNnkQNYn0VqgLevB7uzyfIPvMS7h9YBfp9CpVDLgh1yLchc=","254:UINT":1,"2:UINT":4882,"3:UINT":1,"4:UINT":"89764395371061248","5:STRING":""}]} }
3."4:UINT":"89764395371061248" : the data type is UINT, but we got a string result

Bug prevalence
always
GitHub hash of the SDK that was being used
181b0cb
Platform
android
Platform Version(s)
No response
Type
Platform Issue
Anything else?
No response

the data type is UINT, but we got a string result

That's because the integer is too big to be represented as a number in JavaScript (which only supports integers up to 2^53).
In theory JSON does not have that restriction, but in practice it does.


Minor: rename NestedLoopsJoin memory reservation for clarity

Which issue does this PR close?
Follow on to https://github.com/apache/arrow-datafusion/pull/6188
Rationale for this change
@korowa  pointed out in https://github.com/apache/arrow-datafusion/pull/6188#discussion_r1183282264 that the naming of the memory reservations in NestedLoopsJoin could be improved
What changes are included in this PR?

Change name of variables
Change name of reservation to NestedLoopJoinLoad for the load stream and NestedLoopJoinStream for the output stream

Are these changes tested?
Yes

Are there any user-facing changes?
Not really (slightly different message on failure)

Thank you for the review @r4ntix


Permission denied for public and private dir

I have 2 devices.  On device #1  I am able to use kbfs w/o issue.   On device #2 though I get Permission Denied when I try to access my public or private dir.
ls /keybase/public ls: /keybase/public: Permission denied
my log id: 11871cd8d1c931124bd4d21c

According to the logs, Keybase isn't configured to mount the filesystem.  Have you enabled Finder integration on device 2?  Check the settings tab in the Keybase app.

I have enabled Finder integration am able to browse public and private folder through the Keybase client now.   Still unable however to access via the command line.   Permission denied still popping up

and am able to browse public and private folder through the Keybase client now

You mean you can browse within the "Files" section of the Keybase client?  That doesn't depend on Finder integration, it's just part of our app.  Is there a Keybase favorite available in your Finder that you can browser like a local folder?
If not, try quitting Keybase and relaunching the app, maybe Finder integration hasn't completed yet.

Restarting keybase fixed it.  Able to browse via finder and command line now.   Thanks for your help

cc: @jzila maybe another issue with the Finder integration setting not restarting things anymore.


fix: Remove lodash and replace table

Follow-up of https://github.com/nuxt/consola/pull/20
lodash has a 1.1MB install size and is 60% of webpackbar according to bundlephobia and packagephobia:

Install size: https://packagephobia.now.sh/result?p=lodash
Bundle size<EMAIL_ADDRESS>
Regarding replacing the table dependency:
table is 29.2KB based on<EMAIL_ADDRESS>it also has a 3MB install size and has a dependency on lodash.
text-table has no dependencies and is 11KB in total.
The goal of the PR is just to replace lodash, maybe some of these functions can be removed alltogether

It seems like there's only a check on Windows 🤔

Hi. Thanks for this PR. I really appreciate your help.
Regarding lodash, things are more complicated than just checking this plugin against bundlephobia.
Most of the users are using this package alongside with other dependencies that depend on lodash but may not depend on lodash.* dependencies. So at usual cases we are reusing lodash and i think it would be better to use that dependency.

@timneutkens Really thanks for your contribution. I've removed lodash dependency (at all) and replaced table with text-table as you suggested. WP2 is coming soon :)

:100:


Use refs to classes rather than hardcoded names

It's better to refer to classes and use their names/packages rather than hardcode them and try to not forget while refactoring

Codecov Report

Merging #951 (61e8663) into main (9aa0588) will decrease coverage by 0.28%.
The diff coverage is 100.00%.

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the GitHub App Integration for your organization. Read more.
@@             Coverage Diff              @@
##               main     #951      +/-   ##
============================================
- Coverage     92.29%   92.02%   -0.28%
+ Complexity     2750     2743       -7
============================================
  Files           289      289
  Lines          5490     5491       +1
  Branches        596      597       +1
============================================
- Hits           5067     5053      -14
- Misses          269      278       +9
- Partials        154      160       +6

Files Changed
Coverage Δ

.../java/net/datafaker/service/FakeValuesService.java
84.35% <100.00%> (-0.16%)
:arrow_down:

... and 5 files with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Select doesn't update if something else changes the state

I have something like this:
<dg.Select
                        label="normal map"
                        options={this.props.normalMaps}
                        value={this.props.normalMap}
                        onChange={this.props.onNormalMapChange}
                    />

                    <dg.Button
                        label="Next map"
                        onClick={this.props.onNextNormalMap}
                    />

this.props.normalMap changes as result of the Button's onClick, but the select element does not.

Will give it a shot.

I couldn't actually get it to work, changing it results in the demo only being able to go to Fractal once, and never back to Smooth.

ah crap. just fired up a fresh create-react-app to test and fix this... we'll have to rewrite context usage to support react 16 :)
Trying to reproduce and fix, expect an update soon

Actually it worked, i think it's something with the demo that i found confusing, the props never got triggered. If i change it to state, it works.

I can make a PR.

Not sure what demo we are talking about.. Could you clarify / send link?

example/demo.jsx the the handlers in example/main.js don't trigger rerenders.


Add BonVoyage from SpaceDock

This pull request was automatically generated by SpaceDock on behalf of RealGecko, to add BonVoyage to CKAN.
Mod details:
name = /mod/950/BonVoyage
author = RealGecko
abstract = Make your wheels rolling
license = GPL-3.0
Homepage = http://forum.kerbalspaceprogram.com/index.php?/topic/147876-113-bon-voyage-0999-make-your-wheels-rolling-2016-09-17/
description = ** Preface:
**
What's the best way of getting science from celestial body? The answer is DRIVE!
What's the bast way of delivering ore or supplies from distant location to your base? The answer is make your wheels rolling!
But did you try to drive for 110 km for two hours, avoiding obstacles (screw you Mun Highlands) and keeping you speed in sane limits? Well I did! And that was boooooooring! And who wants to be bored by super crazy explosion simulator called KSP? I guess no one!
Well, no more boredom! Bon Voyage is here and it will drive your rovers in background!
**Story:
**
http://imgur.com/a/YR1WM
**How it works:
**
I don't know.... I honestly don't....
OK OK, just kidding.
Rover must be landed
It has to have BonVoyage controller aboard, seek in Control tab :D
All rover wheels must be intact
At least four of rover's operable wheels must be on the ground
Rover must be landed
Rover will move with average speed equal to 70% of wheels max speed
To start moving your power production must be at least 55% of wheels max consumption
Rover can be solar powered
Rover must be landed
Rover can have alternative power source(RTG)
If rover cannot move without solar power it will idle at night, until Sun is above 5 degrees above the horizon
There must be a clear straight path to target
ROVER MUST BE LANDED!!!

PS: Path finding algorithm is just for demonstration, until I find the way to store 100+ way points in every vessel.
**Bonus:
**
Victims of the alpha test. Not for the faint of heart. Sponsored by HyperEdit(R)
http://imgur.com/a/Zq28v
Please direct questions about this pull request to RealGecko.

Anybody?
I have update already :D

Looks good, merging. Sorry for taking so long


Index graph object model instanceId attribute

Given this is our primary lookup field, we should probably index it.
@RackHD/corecommitters @VulpesArtificem

:+1:

:+1:

👍

test this please

test this please

@benbp @yyscamper
Jenkins encounter memory leak , so GC overhead limit exceeded ,and most job fails .
we re-start jenkins and re-test those PRs.

@benbp I'm not sure this change is necessary.  If the unique property is set for a model attribute, it should automatically be indexed by sails-mongo: https://github.com/balderdashy/sails-mongo/blob/62a3ca0a1d6e3088f420c1a606b643a6dae1296d/lib/collection.js#L407.  Also we probably want to be setting the primaryKey property.

@benbp Actually, I don't think this works at all if we have our default migration strategy set to safe.  See https://github.com/balderdashy/sails-mongo/issues/290.  I think we need to manually create indexes by adding a `setIndexes class method to the graph object model similar to what is done for lookups and obms.

@brianparry ah! Ok, well in that case I'll re-open https://github.com/RackHD/on-taskgraph/pull/167, if you don't mind reviewing.


[Bug] 1.78.0  bridge_generated.io.dart - Can't have modifier 'final' here

Describe the bug

Codegen logs with RUST_LOG=debug environment variable
Can't have modifier 'final' here.

To Reproduce
No response
Expected behavior
No response
Generated binding code
No response
OS
win 10 lstsc 2019
Version of flutter_rust_bridge_codegen
1.78.0
Flutter info
No response
Version of clang++
No response
Version of ffigen
8.0.2
Additional context
No response

Could you please give a minimal reproducible sample?
In addition, I guess your Flutter version is too low - need 3.x

Ok, this needs to be changed   pubspec.yaml

@107295472 what did you change?

@107295472 what did you change?

environment:
sdk: '>=3.0.0'


Pass Data

Is it possible to pass data to DemoPopupViewController like a string or nsdata?

Yes, If a parentViewController has a DemoPopupViewController instance, you can pass any data to it.
Or, if you don't want to have an instance, use NotificationCenter or something like that.
Thank you.


Exercises/01-azure-search.html - Could not execute instructions via Visual Studio code but found workaround

For context, I tried to do this with the following system:
iMac 2020 on Sonoma 14.4.1
I made sure Azure CLI, Powershell, and the Python packages were installed, my Azure storage and AI search were all in the same Azure multi-services resource group, and I used the Storage data for configuration of authorization to my "UploadDocs.cmd" file
Upload Documents to Azure Storage
Per the exercise instructions, Step 5 states, after syncing one's Azure CLI to Visual Code, to run the following command to upload the soon-to-be used documents for the experiment in using Azure AI Search:
UploadDocs
I couldn't get that to execute in either my zsh shell or Powershell via Visual Code... I tried, "Invoke-Command [file-path\UploadDocs.cmd]", "Invoke-Item [file-path\UploadDocs.cmd]", ".\UploadDocs.cmd", or simply, "UploadDocs.cmd" and on zsh I tried, "sh UploadDocs.cmd", to no avail with all of the above.
My Workaround - Execute Azure CLI commands directly in the terminal/zsh
I probably tried some more, but I can't remember. I'm not sure if this is an error but the workaround that I ended up using was to copy/paste the 'call' instructions from the UploadDocs.cmd file and replace the fields: "!azure_storage_account!", "!subscription_id!", and "!azure_storage_key!"
with the corresponding authorization data for my storage account.
Thereafter, I executed the commands in my zsh terminal (with Azure CLI installed and logged into, of course) and when I checked my portal, I saw the container added with the first command executed and when uploading the data for the lab, I saw the batch being uploaded and confirmed as completed via the terminal shell itself. Hope it helps, and also would love to hear what the optimal solution was that was originally intended for the lab. I imagine same can be done similarly with Powershell for Windows users once Azure CLI is installed but considering I'm on a Mac, I switched back to use of zsh when executing the Azure CLI commands.

My solution that worked:
Convert the script to a Bash Shell Script:
#!/bin/bash

# Set values for your storage account
subscription_id="YOUR-SUBSCRIPTION-ID"
azure_storage_account="STORAGE-ACCOUNT-NAME"
azure_storage_key="STORAGE-KEY"

# Create container
echo "Creating container..."
az storage container create \
  --account-name "$azure_storage_account" \
  --subscription "$subscription_id" \
  --name margies \
  --public-access blob \
  --auth-mode key \
  --account-key "$azure_storage_key" \
  --output none

# Upload files
echo "Uploading files..."
az storage blob upload-batch \
  -d margies \
  -s data \
  --account-name "$azure_storage_account" \
  --auth-mode key \
  --account-key "$azure_storage_key" \
  --output none

THEN RUN FILE USING: bash UploadDocs.sh
Let me know if it help


Unexpected E0F while looking for matching

I get the following error after installing Gloss:
`PhaseScriptExecution [CP]\ Embed\ Pods\ Frameworks /Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Intermediates.noindex/REEDSSC.build/Debug-iphonesimulator/REED'S\ S&C.build/Script-F4EFDE5711B38ED5261C982E.sh
cd "/Users/georgedendle/Dropbox/REED'S S&C"
/bin/sh -c "/Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Intermediates.noindex/REEDSSC.build/Debug-iphonesimulator/REED'S\ S&C.build/Script-F4EFDE5711B38ED5261C982E.sh"
mkdir -p /Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Products/Debug-iphonesimulator/REED'S S&C.app/Frameworks
rsync --delete -av --filter P .*.?????? --filter "- CVS/" --filter "- .svn/" --filter "- .git/" --filter "- .hg/" --filter "- Headers" --filter "- PrivateHeaders" --filter "- Modules" "/Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Products/Debug-iphonesimulator/Gloss/Gloss.framework" "/Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Products/Debug-iphonesimulator/REED'S S&C.app/Frameworks"
building file list ... done
Gloss.framework/
Gloss.framework/Gloss
Gloss.framework/Info.plist
sent 309983 bytes  received 70 bytes  620106.00 bytes/sec
total size is 309714  speedup is 1.00
Code Signing /Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Products/Debug-iphonesimulator/REED'S S&C.app/Frameworks/Gloss.framework with Identity -
/usr/bin/codesign --force --sign -  --preserve-metadata=identifier,entitlements '/Users/georgedendle/Library/Developer/Xcode/DerivedData/REEDSSC-ehlgmzzgrccyvbhcpfucoufqunar/Build/Products/Debug-iphonesimulator/REED'S S&C.app/Frameworks/Gloss.framework'
/Users/georgedendle/Dropbox/REED'S S&C/Pods/Target Support Files/Pods-REED'S S&C/Pods-REED'S S&C-frameworks.sh: eval: line 113: unexpected EOF while looking for matching '' Command /bin/sh failed with exit code 2

that doesn’t indicate a problem with Gloss, rather perhaps CocoaPods or Xcode configuration.

How would you fix this? As it isn't appearing for my other pods installed


Running functions from MOD files via LLVM JIT

This is a draft PR to gather some feedback and suggestions :)
It introduces:

A new JITDriver class that, given a LLVM IR module, set ups the JIT compiler and is able to look up a function and execute it. Currently, the result of the execution is printed to stdout and can be a double only.
nmodl_llvm_runner executable that can take a mod file and a specified entry-point function, and run it via LLVM code generation pipeline and  JITDriver.

For example, for MOD file like
// foo.mod
FUNCTION one() {
    one = 1
}

FUNCTION bar() {
    bar = one() + exp(1)
}

running nmodl_llvm_runner -f foo.mod -e bar gives
Result: 3.718282

Current status:
[x] An executable to run functions from MOD files
[x] JIT class to wrap around LLVM's JIT, which can be used in executable(mentioned above) or testing
[ ] Function results and optimisations tests
fixes #482

@georgemitenkov : which version of cmake-format you have? it should be 0.6.0. For this PR, on my machine I see:
± |georgemitenkov/llvm-jit {2} U:1 ✗| → cmake-format --version
0.6.0

± |georgemitenkov/llvm-jit {2} U:1 ✗| → make cmake-format
Built target cmake-format

± |georgemitenkov/llvm-jit {2} U:2 ✗| → git diff
diff --git a/ext/pybind11 b/ext/pybind11
--- a/ext/pybind11
+++ b/ext/pybind11
@@ -1 +1 @@
-Subproject commit f1abf5d9159b805674197f6bc443592e631c9130
+Subproject commit f1abf5d9159b805674197f6bc443592e631c9130-dirty
diff --git a/src/codegen/llvm/CMakeLists.txt b/src/codegen/llvm/CMakeLists.txt
index e8d4296..8cfc269 100644
--- a/src/codegen/llvm/CMakeLists.txt
+++ b/src/codegen/llvm/CMakeLists.txt
@@ -3,8 +3,7 @@
 # =============================================================================
 set(LLVM_CODEGEN_SOURCE_FILES
     ${CMAKE_CURRENT_SOURCE_DIR}/codegen_llvm_visitor.cpp
-    ${CMAKE_CURRENT_SOURCE_DIR}/codegen_llvm_visitor.hpp
-    ${CMAKE_CURRENT_SOURCE_DIR}/jit_driver.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/codegen_llvm_visitor.hpp ${CMAKE_CURRENT_SOURCE_DIR}/jit_driver.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/jit_driver.hpp)

 # =============================================================================
@@ -41,6 +40,5 @@ endif()
 # =============================================================================

 if(NOT NMODL_AS_SUBPROJECT)
-  install(TARGETS nmodl_llvm_runner
-          DESTINATION ${NMODL_INSTALL_DIR_SUFFIX}bin/llvm)
+  install(TARGETS nmodl_llvm_runner DESTINATION ${NMODL_INSTALL_DIR_SUFFIX}bin/llvm)
 endif()

@georgemitenkov : which version of cmake-format you have? it should be 0.6.0. For this PR, on my machine I see:
± |georgemitenkov/llvm-jit {2} U:1 ✗| → cmake-format --version
0.6.0

± |georgemitenkov/llvm-jit {2} U:1 ✗| → make cmake-format
Built target cmake-format

± |georgemitenkov/llvm-jit {2} U:2 ✗| → git diff
diff --git a/ext/pybind11 b/ext/pybind11
--- a/ext/pybind11
+++ b/ext/pybind11
@@ -1 +1 @@
-Subproject commit f1abf5d9159b805674197f6bc443592e631c9130
+Subproject commit f1abf5d9159b805674197f6bc443592e631c9130-dirty
diff --git a/src/codegen/llvm/CMakeLists.txt b/src/codegen/llvm/CMakeLists.txt
index e8d4296..8cfc269 100644
--- a/src/codegen/llvm/CMakeLists.txt
+++ b/src/codegen/llvm/CMakeLists.txt
@@ -3,8 +3,7 @@
 # =============================================================================
 set(LLVM_CODEGEN_SOURCE_FILES
     ${CMAKE_CURRENT_SOURCE_DIR}/codegen_llvm_visitor.cpp
-    ${CMAKE_CURRENT_SOURCE_DIR}/codegen_llvm_visitor.hpp
-    ${CMAKE_CURRENT_SOURCE_DIR}/jit_driver.cpp
+    ${CMAKE_CURRENT_SOURCE_DIR}/codegen_llvm_visitor.hpp ${CMAKE_CURRENT_SOURCE_DIR}/jit_driver.cpp
     ${CMAKE_CURRENT_SOURCE_DIR}/jit_driver.hpp)

 # =============================================================================
@@ -41,6 +40,5 @@ endif()
 # =============================================================================

 if(NOT NMODL_AS_SUBPROJECT)
-  install(TARGETS nmodl_llvm_runner
-          DESTINATION ${NMODL_INSTALL_DIR_SUFFIX}bin/llvm)
+  install(TARGETS nmodl_llvm_runner DESTINATION ${NMODL_INSTALL_DIR_SUFFIX}bin/llvm)
 endif()

cmake-format

That's the issue then! I have 0.6.13

cmake-format

That's the issue then! I have 0.6.13

@georgemitenkov  : I see following error in CI :
/jenkins/05/workspace/hpc.nmodl/platform/bb5/src/codegen/llvm/jit_driver.cpp(38): error: no suitable user-defined conversion from "lambda [](llvm::orc::JITTargetMachineBuilder)->llvm::Expected<std::unique_ptr<<error-type>, std::default_delete<<error-type>>>>" to "llvm::orc::LLJITBuilderState::CompileFunctionCreator" exists
          llvm::orc::LLJITBuilder().setCompileFunctionCreator(compile_function_creator).create());
                                                         ^

Which LLVM version you are using? master or 11.0 ?  If it's hassle to be compatible with 9.0, we can just drop it and move to 11.0. I can update our Jenkins CI.

@georgemitenkov  : I see following error in CI :
/jenkins/05/workspace/hpc.nmodl/platform/bb5/src/codegen/llvm/jit_driver.cpp(38): error: no suitable user-defined conversion from "lambda [](llvm::orc::JITTargetMachineBuilder)->llvm::Expected<std::unique_ptr<<error-type>, std::default_delete<<error-type>>>>" to "llvm::orc::LLJITBuilderState::CompileFunctionCreator" exists
          llvm::orc::LLJITBuilder().setCompileFunctionCreator(compile_function_creator).create());
                                                         ^

Which LLVM version you are using? master or 11.0 ?  If it's hassle to be compatible with 9.0, we can just drop it and move to 11.0. I can update our Jenkins CI.

I am using LLVM 11. I think it makes sense to upgrade to 11 on CI? As far as I am aware LLVM tries to keep up to date with new C++ standards, so lambdas, callbacks, etc. are used more and more frequently.
If that's a problem, I can try to rewrite it in the more standard way too.

I am using LLVM 11. I think it makes sense to upgrade to 11 on CI? As far as I am aware LLVM tries to keep up to date with new C++ standards, so lambdas, callbacks, etc. are used more and more frequently.
If that's a problem, I can try to rewrite it in the more standard way too.

If that's a problem, I can try to rewrite it in the more standard way too.

I would upgrade to v11.0 in CI for two reasons:

We don't expect end users of NMODL to build against LLVM (this will be pre-build package)
LLVM API changes (often) and I don't see need to spend efforts to make it backward compatible

I will update CI shortly and re-trigger build.

If that's a problem, I can try to rewrite it in the more standard way too.

I would upgrade to v11.0 in CI for two reasons:

We don't expect end users of NMODL to build against LLVM (this will be pre-build package)
LLVM API changes (often) and I don't see need to spend efforts to make it backward compatible

I will update CI shortly and re-trigger build.

Retest this please

Retest this please

@pramodk I have rebased on your recent PR, so this should be ready for the merge now!
I have made some additions to the previous draft version, particularly:

Now there is a Runner class that wraps around JIT driver. It helps to initialise JIT with LLVM IR module only once, and then we can run multiple functions from it. For example:

// Create runner only once
Runner runner(...);

runner.run("foo");
runner.run("bar");
...

Set up framework for some other return types with templates, but currently we only use doubles.
Added some tests to check the results of executing LLVM IR module functions
Added tests to verify optimisation does not break anything. This may be changed if we change the way we optimise our IR.

@pramodk I have rebased on your recent PR, so this should be ready for the merge now!
I have made some additions to the previous draft version, particularly:

Now there is a Runner class that wraps around JIT driver. It helps to initialise JIT with LLVM IR module only once, and then we can run multiple functions from it. For example:

// Create runner only once
Runner runner(...);

runner.run("foo");
runner.run("bar");
...

Set up framework for some other return types with templates, but currently we only use doubles.
Added some tests to check the results of executing LLVM IR module functions
Added tests to verify optimisation does not break anything. This may be changed if we change the way we optimise our IR.


Question: Is Paket available as a library?

Description
Is Paket available as a library?
Repro steps
I tried the following:
source https://api.nuget.org/v3/index.json

storage: none
framework: net5.0, netstandard2.1

nuget FSharp.Core
nuget Paket

But I think this fetches Paket the tool not a library.
Expected behavior
I would like to use the Paket code as a library.
Actual behavior
paket install succeeds, but dontnet build fails.
error NU1202: Package Paket 5.257.0 is not compatible with net5.0 (.NETCoreApp,Version=v5.0).
Package Paket 5.257.0 supports: netcoreapp2.1 (.NETCoreApp,Version=v2.1) / any

error NU1212: Invalid project-package combination for Paket 5.257.0.
DotnetToolReference project style can only contain references of the DotnetTool type

(Note I also tried setting my project to netcoreapp2.1)
Known workarounds
Git Submodules?

Have you tried Paket.Core?

Yes that works. Thanks!


Add new test cases for RAddCommandParser

Resolves #353

Codecov Report

Merging #354 (9caa8e9) into master (1189235) will increase coverage by 0.09%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #354      +/-   ##
============================================
+ Coverage     72.44%   72.54%   +0.09%
  Complexity      680      680
============================================
  Files           101      101
  Lines          2116     2116
  Branches        242      242
============================================
+ Hits           1533     1535       +2
+ Misses          545      543       -2
  Partials         38       38

Impacted Files
Coverage Δ

...java/foodwhere/logic/parser/RAddCommandParser.java
100.00% <0.00%> (+8.69%)
:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


broken PayPal button image

Thanks for AlaSQL, looking forward to using it at some point :)

Something not working as expected:

Describe the problem  : broken paypal button image

Thank you for reaching out.
Is the image still missing? When I open the page I get:

works now, strange, and I had hard-refreshed too. cheers!


Adding TeslaCustomComponent

This is created by mlowijs & me. Fixes the wake up which doesnt work in the default tesla component

I can not verify the contributions (github will not show me)
@mlowijs Are you OK with adding this?

Yes, that is fine.
From: Joakim Sørensen<EMAIL_ADDRESS>Reply to: custom-components/hacs<EMAIL_ADDRESS>Date: Friday, 19 July 2019 at 08:00
To: custom-components/hacs<EMAIL_ADDRESS>Cc: Michiel Lowijs<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [custom-components/hacs] Adding TeslaCustomComponent (#300)
I can not verify the contributions (github will not show me)
@mlowijshttps://github.com/mlowijs Are you OK with adding this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/custom-components/hacs/pull/300?email_source=notifications&email_token=AAU7XPIHXQXEEA6IULA2LOTQAFKATA5CNFSM4IE7ZLZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2KVEEY#issuecomment-513102355, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAU7XPJFS32DELBOFGIQP4DQAFKATANCNFSM4IE7ZLZQ.


Exception: Failed to activate node delta_w

Currently running the adaptive HyperNEAT example using device = "cuda:0"
Population size: 100
Elitism: 10%
No multiprocessing
It runs okay for several generations, up until the following exception is thrown from the cppn.py
Any help to sort out this issue is appreciated.
 File "/pytorch_neat/cppn.py", line 108, in __call__
    return self.get_activs(shape)
  File "/pytorch_neat/cppn.py", line 93, in get_activs
    self.activs = self.activate(xs, shape)
  File "/pytorch_neat/cppn.py", line 87, in activate
    raise Exception("Failed to activate node {}".format(self.name))
Exception: Failed to activate node delta_w

Additional info on Traceback : RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
Could this be because self.aggregation(inputs) is not correctly implemented for cuda:0 device?
Traceback (most recent call last):
  File "/pytorch_neat/cppn.py", line 83, in activate
    pre_activs = self.aggregation(inputs)
  File "/pytorch_neat/aggregations.py", line 20, in sum_aggregation
    return sum(inputs)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Thank you in advance.


Tests: bluetooth: tester: find connection for get_attr_val_rp

To check security of connection  in read_value() we need to actually
pass connection not NULL.
This is affecting GATT/SR/GAR/BI-11-C
Signed-off-by: Krzysztof Kopyściński<EMAIL_ADDRESS>

FYI @sjanc @michal-narajowski


[ENH] Add links to example datasets for each modality

Fixes #842
Add links to example from the BIDS example repo in the following section:

[x] MEG
[x] iEEG
[x] genetics
[x] PET
[ ] MRI

[ ] anat
[ ] func
[x] dwi
[x] fmap
[x] asl
[ ] qmri

[ ] behavior ?
[x] physio

While working on this I am tempted to split the table in the README of BIDS-example to match better with the BEP:
https://github.com/bids-standard/bids-examples#dataset-index

get the PET example out of the multomodal table
get the qMRI and the ASL examples out of the MRI dataset and in their own respective tables

I would be easier to then have a single link to those sub-tables in the spec, than a whole list of links in the spec. Also this way we can add more examples in BIDS example without having to do the same in the spec.


Deadlocks stacking and mini-loaders causing error

Mod source
Factorio Mod Portal
Which mod are you having an issue with?

[ ] pyalienlife
[ ] pyalternativeenergy
[ ] pycoalprocessing
[ ] pyfusionenergy
[ ] pyhightech
[ ] pyindustry
[ ] pypetroleumhandling
[X] pypostprocessing
[ ] pyrawores

Operating system

=Windows 10

What kind of issue is this?

[ ] Compatibility
[ ] Locale (names, descriptions, unknown keys)
[ ] Graphical
[ ] Crash
[ ] Progression
[ ] Balance
[X] Pypostprocessing failure
[ ] Other

What is the problem?

Steps to reproduce
Error occurs on loading with both deadlocks and post processing enabled.
Additional context
No response
Log file
No response

This is fixed for the next release. Thanks for the report.


Crashes without error message

z_i,* = x_i,* × y_i,*

where
x ∈ ℝ^(n×3)
y ∈ ℝ^(n×3)

I would have thought this would set the ith row of z to be the cross product of the ith rows of x and y.
If this is not valid I❤LA, then we should at least output a parse error. Currently this crashes the compiler:
Exception: ('Unexpected type %s walked', 'str')

This is not valid in I❤LA now, the crash should be fixed


[Accessibility] Make editor.action.toggleTabFocusMode configurable in user settings

CC @meganrogge
Currently, the state of editor.action.toggleTabFocusMode is not preserved and users need to toggle this on and off manually each time they reload VSCode.
Two discussion points are:

What if users want to preserve this state?

What if users want to have different tab key behaviors between editor and terminal? For example, I prefer to toggle off tab move focus in editor whereas I would like to keep it on in terminal.

Expected Behaviors
It would be great if users could configure the tab key behavior in editor and terminal context separately.
System Info
Version: 1.76.0-insider (system setup)
Commit: c4a609f94329b17d91cd13a4fd113d1abfe5ec1f
Date: 2023-02-03T05:23:17.202Z
Electron: 19.1.9
Chromium: 102.0.5005.194
Node.js: 16.14.2
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621
Sandboxed: Yes

This feature request is now a candidate for our backlog. The community has 60 days to upvote the issue. If it receives 20 upvotes we will move it to our backlog. If not, we will close it. To learn more about how we handle feature requests, please see our documentation.
Happy Coding!

A downside of having this as a setting is your settings can change a lot. Maybe the best of both worlds is something like the command toggles it without changing the setting, and the setting defines the initial state when the editor loads?

A downside of having this as a setting is your settings can change a lot.

What do you mean by this? @Tyriar
I believe the toggling action was likely introduced to account for operating in different contexts IE the terminal vs editor. It makes sense that a user would always want it on/off for each context and not want to have to toggle it.

I believe it's a state that a user would typically toggle on and off frequently, settings.json is meant to be a more stable and not change so much.

Yep, let's keep the command to toggle and add separate settings for the terminal and workbench so it persists on reload and doesn't require toggling between the different components

I think this action should be available via the command palette

this is very involved if we want the status bar item to reflect the persisted state/ setting value as the linked commit shows

Found a simpler approach


Watch for new conda environments

Before we can expect Jupyter to completely rely on the discovery API in https://github.com/microsoft/vscode-python/issues/19101, add this fix from Jupyter to here. It watches environments.txt so isn't that expensive.

Verification steps:

Install pre-release of extension
Activate the extension using Python: Select Interpreter command, wait for the list to load.
Create conda environment using any of the 4 commands:

conda create -n name1 python
conda create -p . python
conda create -n name2
conda create -p .

Open the list again make sure it appears.


API incorrectly shows Walking Wake learns fake-out

Walking Wake doesn't learn fake out though.

Hi can you fix it in our CSV?

Hi can you fix it in our CSV?

Sounds good!


Vickel: Watch svarna.co

Vickel requests the watch of the watch_keyword svarna\.co. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

svarna\.co has been seen in 1 true positive, 0 false positives, and 0 NAAs.

Approved by Jeff Schaller in Charcoal HQ


Question/bug? about assume negative loss function

If i have read the paper and code correctly. Unobserved labels should be labelled as -1 and observed labels should be 1.  In the function 'loss_an' it checks for labels that are zero while in the paper the formula says !=1 which is a bug in the code I think as it will only consider positive observed samples causing each label to always be activated while training.

Thanks for the question!
The notation in the paper does differ from the implementation in the code. I think the confusion stems from a (probably sub-optimal) choice I made to use different label representations for fully observed losses and SPML losses in the code.
For fully observed losses like $\mathcal{L}_\mathrm{BCE}$ we use +1 for positive and 0 for negative. To see how we format fully observed label matrices, see e.g. here.
For partially observed losses like $\mathcal{L}_\mathrm{AN}$ we use +1 for positive, -1 for negative, and 0 for unobserved. For SPML in particular we expect one label to be positive (+1) and the rest to be unobserved (0). We should not have any observed negative labels (-1). To see how we convert fully observed label matrices to observed label matrices, see here.
Please re-open this issue if I have not answered your question!

Thank you so much for the quick response!  This actually helps me a lot thank you


Bug: GitHub Action deploy frequently fails and takens 20 minutes

Describe the bug
There have been a number of failures in the GitHub actions recently when deploying the web app. Strangely, it does seem to work but the action reports and error, suggesting to look at the diagnostics logs even though there doesn't appear to be any.
https://github.com/AdmiralGT/botc-scripts/actions

The problem here is with the collectstatic django call which doesn't appear to be able to cache all the fontawesomefree content. I've temporarily turned off collecting of static data and I'll turn it back on when I need to add static data since I rarely add new static data.


Plugin upgrades

deprecate prism-languages in favor of require('prismjs').languages
fix json syntax highlighting by using native json highlighter
automatically load syntax definitions from prismjs/components folder
upgrade Prism.js to v1.5.1
Closes #6. Ability to introduce custom styles for syntax highlighting in book.json.   Defaults to prismjs/themes/prism.css

"pluginsConfig": {
  "prism": {
    "css": [
      "prismjs/themes/prism-solarizedlight.css"
    ]
  }
}

@gaearon this PR seems cool, could you please take a look?

I added @robmcguinness to collaborators here and on npm.
@robmcguinness Please feel free to merge and cut a release!

Thanks @gaearon.  I should be able to publish a new version today.


Barback should preserve laziness when lazy assets are piped through dry-run transformers

 Issue by nex3
Originally opened as dart-lang/sdk#16442

Barback currently materializes lazy assets as soon as they're passed to a new phase, so effectively only final-phase transformers can be lazy. It should preserve laziness as long as the assets are only piped through dry-run transformers.

 Comment by lrhn

Removed Area-Library label.
Added Area-Pkg label.

 Comment by anders-sandholm

Removed Library-Barback label.
Added Pkg-Barback label.

 Comment by nex3

Fixed by dart-lang/sdk@ff12f81d44a246fc8795820cb26459869c4816e9.

Added Fixed label.

 Comment by Sfshaza

Marked this as blocking dart-lang/sdk#16684.

 Comment by nex3

Marked this as blocking dart-lang/sdk#17903.

 Comment by nex3

Added Started label.


Rev DevTools to 2.1.1 for tests

https://github.com/flutter/flutter/issues/81513

Will submit on red flutter-build after all checks pass.


.gifs don't work on image upload once processed

Context
The initial preview of a gif shows as expected but after it is processed by the lambda it loses the motion.
Expected Behavior
gifs can be resized but should still work.

Oh wow.... my gif people need me!
Can you document the flow of how an image gets resized @NiallJoeMaher  or @John-Paul-Larkin ?
Is it

User adds image to an article
Image gets sent to lambda to be resized ?
If image is a gif its now broken?

Yes, you'll see it in the CDK lambdas folder. From memory it's just because we convert those gifs to a static image


add goerli to infura networks

allow to use goerli infura testnet with mythril tool

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@norhh


tutorial.ipynb must be tested and roughly functional!!!

I just started looking into that.
The current version of the tutorial is focusing on the Redpitaya class. I wonder if we are not diverting the user's attention from the Pyrpl class that he is encouraged to use instead ?

In particular, there would be a little work involved if we want to bring the Redpitaya class alone up-and-running (some problems right now with the function returning the list of inputs of dsp modules that looks for the root pyrpl instance).
Since all unittests are performed with the full pyrpl instance, I would advocate to use this within the tutorial....
--> I am doing that right now.

agreed. also, i suggest that all installation stuff is moved to the documentation such that the tutorial can focus more on doing actual stuff. also, the tutorial could be separated into several parts by now.

OK, so I should probably touch at the same time the documentation and the tutorial. I think I just succesfully ran the make html for the autodoc. For the moment, do you have anything that is not autogenerated in the documentation ?

documentation is currently distributed over the wiki, the tutorial (former installation section), the readme.md, and the autodoc files.

OK, I guess we want the autodoc to look a bit more organized right ?

The steps towards making the actual documentation are

check https://pythonhosted.org/an_example_pypi_project/sphinx.html and pick the most reasonable docstring format

describe that format on this wiki page: https://github.com/lneuhaus/pyrpl/wiki/Code:-Coding-style-guide#docstrings

for the automatic generation, we could either make a branch 'gh-pages', or just host them in the 'doc' folder

implement automatic autodoc generation, and describe what is done here: https://github.com/lneuhaus/pyrpl/wiki/Packaging:-AutoDoc

see also #170

@SamuelDeleglise
in principle the choice between google-style () and numpy-style (http://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html#example-numpy) is a matter of taste, even though the google-style is "recommended by Khan's academy", whatever that means. I would go with google-style because i find it easier to write (and I believe pycharm can be customized), but i would also be okay with numpy-style if you see any advantages there

Concerning the checklist item "modify the source files for autodoc to get a reasonable organization of the documentation":
This is the main decision to make now. I believe the structure of the documentation can be inspired by other pages, such as this one: https://docs.scipy.org/doc/numpy-1.13.0/user/
I believe the other points can be postponed after that..

Sorry, I was drifting in the various links of your posts...
I am fine with Google style, as long as it is properly formatted by the sphynx extension
To start the discussion, the structure could be (I am not sure where the autodoc goes by the way...):

Hardware installation
Installing Pyrpl
Quickstart Tutorial
Basics of pyrpl --> Where we explain modules and general program organization
Contributing  --> Where we put all the stuff on unittests and git pull requests that are currently in the tutorial

OK, they seem to have a User's guide and a Reference... The user's guide is basically what I sketched above and the 'Reference' is basically the autogenerated doc.
Also, I see that they have actually put the Developer's guide in a 3rd section. Actually the root of their sphynx doc is there;
https://docs.scipy.org/doc/numpy-1.13.0/index.html

So, we could probably have:

User's guide

Hardware installation
Installing Pyrpl
Quickstart Tutorial
Basics of pyrpl

Reference guide

A basic hand-written page that lists the different python modules with a link to their autogenerated doc

Developer's guide

Compililng the FPGA code
Unittests
Code guidelines
Workflow to submit code changes

Are you working on it or should I start ?

go ahead if you like

just pull my latest commit with the sphinx configuration

are you workking on it now?

otherwise ill go ahead and test the structure

I was waiting for your answer and trying to implement the validate_and_normalize unit_test in the mean time. Just let me know when I can start the doc

ok go ahead with the unit test, ill try to generate the doc structure for now

i have to go no, continue if you like on the last commit

Alright, you haven't created the structure yet right ?

its there, dont you see it?

in doc/sphinx/source/index and references

go to doc/sphinx, type 'make html' and check the build directory

Did you push it to develop-0.9.3 ?

Got it thanks

I believe the right solution is that you eventually create an account there:
http://dont-be-afraid-to-commit.readthedocs.io/en/latest/documentation.html
it will compile automatically the documentation for us (and of course, host the website).
We also need to find a solution to store the redpitaya image files and/or list of files. I am not sure if this is supported by readthedocs

There is a largr file storage on github we can use fir that..

pyrpl.readthedocs.io will host the docs
however, im already running into trouble with that website to correctly get pyrpl imported because of the pyqt dependency...

readthedocs is working and the content of the documentation is slowly improving.
Still, our website is not very welcoming at the moment. I suggest we try to get our documentation to look more similar to https://automl.github.io/auto-sklearn/stable/

im leaving this open until we have

[ ] adapted the nicer style and more welcoming startup page from https://automl.github.io/auto-sklearn/stable/ or http://docs.obspy.org/
[ ] finished with the tutorial.ipynb file (now at a different location)
the rest of documentation is covered by #170

I think that is the right location. I think in the meantime there was the idea to convert the tutorial into the module documentation on the website, so i had regarded the tutorial as deprecated. But I agree that it is best to have both, especially if testing and the generation of the displayed notebook output can be automatically done.

Do you want the tutorial only for yesting or for new users as well?

My idea was to test the tutorial that new users will have at their disposal such that no errors are introduced in there during development...


chore: Upgrade the Mac CI task

Description of changes
Builds & tests Mac & iOS platforms on oldest & newest Xcode, using a single job matrixed to run 4 combinations of OS, Xcode, and platform.

Eliminates dependence on CRT builder script, and all of the config for that script is removed.
No need for Github or AWS credentials on this workflow anymore.  AWS credentials may be removed from this repo after merge.
Allows for easy add of new Apple platforms (TV, watch, etc.) in the future.

Note: Update the required CI checks to reflect the new names after this PR merges.
New/existing dependencies impact assessment, if applicable
No new dependencies were added to this change.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

/test

/test


Create an XML Schema for the REST query response

Currently the XML is manually created but an XML Schema would help in reading the results.
Based on todo in #25

XSD is live: http://directory.peppol.eu/files/directory-search-result-list-v1.xsd


Attachment should be marked as unshared when related chat was unshared

How to reproduce
User 1 create chat with file pdf (DIAL RAG or echo)
User1 share chat with file
User2 open shared link
User1 open shared chat
User1 revoke access to this file chat
User1 open Manage Attachment form and find file attached to the chat
Actual result: Chat marked as unshared, arrow is removed from chat icon ->ok
File is still marked as shared : arrow is displayed on the file icon-> bug

Note: file also is not unshared when user1 change model to shared chat
Expected result
File should be shown as unshared ( without arrow on file icon)

Agreed with @sdryapko on daily-status-call that it's expected behavior.


Now onConfirmBtnTap, onCancelBtnTap callback functions will return current context

Solved problem where when writing custom onConfirmBtnTap , popping the alert causes the screen to pop as we are popping top level context.
// In below example the alert stays on screen in macos as the we are popping the top level context. Popping it causes current screen to pop.
QuickAlert.show(
              context: context,
              type: QuickAlertType.error,
              title: "Deny rewards ?",
              cancelBtnText: 'Back',
              showCancelBtn: true,
              showConfirmBtn: true,
              //autoCloseDuration: const Duration(seconds: 10),
              onConfirmBtnTap: () {
// Some custom code
                Navigator.pop(context);
              },
            )

//Solution
// With below code we can catch the result too, thus detecting wheather the closure of alert is done by cancel/ok button or barrier dissmissable.
var result = await QuickAlert.show(
              context: context,
              type: QuickAlertType.error,
              title: "Deny rewards ?",
              cancelBtnText: 'Back',
              showCancelBtn: true,
              showConfirmBtn: true,
              //autoCloseDuration: const Duration(seconds: 10),
              onConfirmBtnTap: (ctx) {
// Some custom code
                Navigator.pop(ctx, true);
              },
            )

Any idea when publisher will push this to pub dev ?

Any update on this ?

Any update on this ?

Hello @abhay-s-rawat
No need to take context into the function as a parameter, we can directly address/use the context of the parent.
Happy Coding!!!

If you try to pop eith parent context the current page gets popped not the dialog. I tested same in macos.


Fix many errors when executing _get_fundamentals on particular tickers

https://github.com/ranaroussi/yfinance/issues/172
https://github.com/ranaroussi/yfinance/issues/173
https://github.com/ranaroussi/yfinance/issues/163
https://github.com/ranaroussi/yfinance/issues/162

Hi @ranaroussi , this pull request works, when I pulled it locally and ran a few scraping rounds. Could you merge it, or is something still missing (I see two quality checks failed).
Thanks!

What is more important: greening 'quality checks' or writing bug-free code?
For the last 2 months, I ran this branch against the 8k NASDAQ tickers on a daily basis and it did not fail once.
Anyway if code quality was truly a concern we, as a discipline, would write software in Ada, not Python. Sorry, it has to be told. :)
I suppose we can make this branch pass 'code checks' but I am not interested enough in Python or its Pythonic concerns to bother improving.
Anyway, thanks for chipping in Michel117.

What is more important: greening 'quality checks' or writing bug-free code?

@ohenley I think the quality checks are there to prevent bugs in code, like the one you introduce here: https://github.com/ranaroussi/yfinance/pull/174/files#diff-69e424ccdc9dbd39c5d517098950e475R518

@ohenley I think the quality checks are there to prevent bugs in code, like the one you introduce here: https://github.com/ranaroussi/yfinance/pull/174/files#diff-69e424ccdc9dbd39c5d517098950e475R518

Hey Rob, thanks for chipping in to help me grow! Now I will help you with the basic stuff, the fundamentals:
"A software bug is an error, flaw or fault in a computer program or system that causes it to produce an incorrect or unexpected result," https://en.wikipedia.org/wiki/Software_bug
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> old_search_str = '"|'
>>> print (old_search_str)
"|
>>> ticker = "ROB"
>>> search_str = '"|'.format(ticker)
>>> print (search_str)
"|

It changes nothing and therefore it is not a 'bug'. Oh, its trash, do not get me wrong, but it is totally valid loose ends good old Python.
Like said before, I just bullied my way to make this codebase work, period. Now Pythonic, Pythoneer, Pythonist, Pythonista like you are welcome to replace the not so 'shiny pipes' using 'shiny pipe coverage' tools.
This PR works as it is tested extensively, daily. This useful repo is slowly becoming a PR festival with the author/s arguably ghosting it.
In that context, if you detect useless code that 'red eye' this PR, why don't you just suggest a fix instead of necromancing an argument with me about the 'soundness' of 'code checker' over a fat finger toy language?

@ohenley do these lines of code do what you intend them to do?
https://github.com/ranaroussi/yfinance/blob/cf926fc9309f2cb317fd2808ef2d322927b4a953/yfinance/base.py#L319-L322


Data payload for UC Browser

I have a problem with UC Browser subscribers.
I have a error: {"errno":104,"message":"Data payload too large, limit is 2048 bytes, current is 3070 bytes","code":413}.
Can you fix it?
Thanks.

I have same problem in 2 versions of the library - ^5.1 and ~4.0


Test for packed now binary

Packed executable may fail if not all the files are taken into the executable. It concerns all the dependencies as well as now itself. Such a failure is revealed by simply running the executable - one gets Can not require some-package.

Btw look - i made it run on CI only as it adds at least 10 seconds to npm test

🙏


The first JSX property should always be placed on a new line if the JSX tag takes up multiple lines and there are multiple properties (react/jsx-first-prop-new-line)

Ensure correct position of the first property.
multiline-multiprop: The first property should always be placed on a new line if the JSX tag takes up multiple lines and there are multiple properties.
https://github.com/yannickcr/eslint-plugin-react/blob/master/docs/rules/jsx-first-prop-new-line.md
I've personally run into this while working on WebTorrent Desktop and Brave with @dcposch, where we've been very inconsistent with this.

IMHO:
OK:
<Hello personal={true} />
<Hello foo={{
  x: 1,
  ...
}} />

<Hello
    personal={true}
    foo="bar"
/>

Bad:
<Hello
    personal />

<Hello personal
    prop />

<Hello personal={true}
    foo="bar"
/>

Yeah, that looks right @dcousens!

Just ran the test suite. No errors, except on webtorrent-desktop, and I agree that all the things it caught should be changed. So this LGTM

This will ship in standard 14.


FIX - removed sample files and changed docker.for.mac to host.docker.internal

This PR removes unused docker files (sample files) and changes our usage of docker.for.mac.host.internal to host.docker.internal due to the former's deprecation.
see: https://docs.docker.com/docker-for-mac/release-notes/#docker-community-edition-18030-ce-mac59-2018-03-26

 networking set up on linux host machine


Custom FancyZones layouts missing after restart

Microsoft PowerToys version
0.47.0
Running as admin

[x] Yes

Area(s) with issue?
FancyZones
Steps to reproduce
Uploading PowerToysReport_2021-10-06-09-42-15.zip…
✔ Expected Behavior
Custom layouts remain after restart
❌ Actual Behavior
Custom layouts all missing after restart
Other Software
No response

Duplicate of #13625

Thanks for reporting this. This issue is a duplicate. We'll track this in #13625 to centralize the discussion thread.


make command help display consistent

When I try to use ctr to utilize containerd,
I found there is a little bit improvement I could do to make display consistent.
Signed-off-by: Allen Sun<EMAIL_ADDRESS>

Codecov Report

Merging #1544 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1544   +/-   ##
=======================================
  Coverage   42.61%   42.61%
=======================================
  Files          24       24
  Lines        3318     3318
=======================================
  Hits         1414     1414
  Misses       1581     1581
  Partials      323      323

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 709d34f...9d16eaf. Read the comment docs.

LGTM


[Data Normalization] Fix memory exhausting issue

to: @jacknagz
cc: @airbnb/streamalert-maintainers
size: medium
resolves #654
Background
The fix to allow one record to be normalized multiple times with different set of datatypes in PR #655 causes memory exhausting issue. We can reproduce the issue by profiling memory usage using library memory_profile.
The root cause of memory exhaustion is caused by one record is normalized multiple times with same set of datatypes and normalized records list is grown in linear speed when threat intel feature enabled. User may not notice the memory issue unless there are thousands records been normalized in one lambda invocation.
The solution is to check if a record has been normalized by the same set of datatypes or not. If so, use normalized information from existing normalized records, otherwise apply normalization to the record.
Changes

Extract Data Normalization and Threat Intel logic from process method.
Check alert duplication.
Add helper functions to prevent a record being normalized multiple times by the same set of datatypes.

Testing

Testing environment is locally:

A json file contains 7622 logs. Majority of logs are related to system processes. So the datatypes command, fileHash are heavily applied to normalize records.

One record can  only be normalized once

the memory usage is 52.6MB
Generated 7622 normalized records

Line #    Mem usage    Increment   Line Contents
================================================
    68     85.8 MiB     85.8 MiB       @profile
    69                                 def run(self, event):
    70                                     """StreamAlert Lambda function handler.
  ...
   111                                         # Cache the log sources for this service and entity on the classifier
   112     87.0 MiB      0.0 MiB               if not self.classifier.load_sources(service, entity):
   113                                             continue
   114
   115                                         # Create the StreamPayload to use for encapsulating parsed info
   116     87.0 MiB      0.0 MiB               payload = load_stream_payload(service, entity, raw_record)
   117     87.0 MiB      0.0 MiB               if not payload:
   118                                             continue
   119
   120    120.6 MiB     33.6 MiB               payload_with_normalized_records.extend(self._process_alerts(payload))
   121
...
   159    121.2 MiB      0.0 MiB           if self._firehose_client:
   160    139.6 MiB     18.4 MiB               self._firehose_client.send()
...
   164    139.6 MiB      0.0 MiB           return self._failed_record_count == 0

After merge PR #655, one record can be normalized multiple time

The memory usage will be increased to 1,227.5 MB!! (1.2GB)
Generated 110,483 normalized records

Line #    Mem usage    Increment   Line Contents
================================================
    68     86.0 MiB     86.0 MiB       @profile
    69                                 def run(self, event):
...
    116     86.7 MiB      0.0 MiB               payload = load_stream_payload(service, entity, raw_record)
    117     86.7 MiB      0.0 MiB               if not payload:
    118                                             continue
    119
    120   1290.9 MiB   1204.2 MiB               payload_with_normalized_records.extend(self._process_alerts(payload))
...
    159   1297.4 MiB      0.0 MiB           if self._firehose_client:
    160   1314.2 MiB     16.8 MiB               self._firehose_client.send()
...
    164   1314.2 MiB      0.0 MiB           return self._failed_record_count == 0

Apply current fix which is only allow a record to be normalized multiple times by the different set of datatypes

The memory usage looks reasonable, 143.4MB
Generated 27,534 normalized records

Line #    Mem usage    Increment   Line Contents
================================================
    68     86.0 MiB     86.0 MiB       @profile
    69                                 def run(self, event):
...
   116     87.2 MiB      0.0 MiB               payload = load_stream_payload(service, entity, raw_record)
   117     87.2 MiB      0.0 MiB               if not payload:
   118                                             continue
   119
   120    212.6 MiB    125.5 MiB               payload_with_normalized_records.extend(self._process_alerts(payload))
...
   159    218.8 MiB      0.0 MiB           if self._firehose_client:
   160    230.6 MiB     11.8 MiB               self._firehose_client.send()
...
   164    230.6 MiB      0.0 MiB           return self._failed_record_count == 0
...

Runtime profiling: No noticeable running time changed.

Coverage increased (+0.1%) to 95.695% when pulling 2864c21973e74f91704155ec6999bfaa743a9a92 on fix_data_normalization_exhausts_memory_issue into e342c7a7166bf4e5c1e5f15cc98c0a8ee0b128d7 on master.

@jacknagz Thanks for the comments. PTAL.


Add finder to "links" in all content items

Trello: https://trello.com/c/uR7v1Rri
Depends on: https://github.com/alphagov/info-frontend/pull/444
What
Allows "finders" to be added to the links hash of content items. At moment this is only needed on the business finder. Most people land on content, but then have no way of easily finding other Brexit related content that is relevant to their organisation.
Why
If a piece of content knows about the finders it is surfaced by, then we can use this information to create a link back to the finder on page in the breadcrumbs and related links.

@steventux I've updated the example. Can you re-review?


transformSchema config

this new configuration allows for easy schema transformation before the gqty client is generated
closes #518

Not sure about the actual cause of the original issue though.

it should be some limitation in case where a GraphQL Schema is very big it makes a big typescript file, but it's weird anyways, because GQty is being tested with the GitHub API, which is very big, and everything works fine


various updates for heroku, aws, etc

Current coverage is 15.09% (diff: 0.00%)

Merging #16 into master will not change coverage

@@             master        #16   diff @@
==========================================
  Files            18         18
  Lines           159        159
  Methods          62         62
  Messages          0          0
  Branches         47         47
==========================================
  Hits             24         24
  Misses          135        135
  Partials          0          0

Powered by Codecov. Last update 640c955...ccadd05

PS I added aws environment variables to torus. You can see them using torus view in the project directory


Production issue

It's working great in the dev mode but in production, not sure why it's so weird. FYI, i'm using material-ui

me too


digit..& should be allowed in link params

Ref:
https://github.com/markdown-it/markdown-it/issues/661

ups... should be in linkifier repo


[post-land] propagate diem name change

Motivation
Propagate name changes to post-land test.
Test Plan
Canary https://github.com/diem/diem/actions/runs/421413195

/canary

/canary

/land

:broken_heart: Test Failed - Build images and run cluster test

/land