MosaicML-Streaming on Databricks

Hi all, I'm a new user of mosaicml-streaming on Databricks who stumbled upon Mosaic ML (and Petastorm) for loading large data from PySpark to PyTorch tensors. Here is an example jupyter notebook that I'm trying to replicate on my Databricks clusters, however, I have a few questions:

The above notebook's requirements say that we need a "Databricks Runtime for ML 15.2 or higher". However, my organization has an earlier version. Can we use mosaic-streaming on earlier runtime versions?

This above notebook imports "from petastorm import TransformSpec" but this blog says that Petastorm is deprecated and suggests the usage of mosaic-streaming instead. I checked the code, and it just imports the petastorm but doesn't use it. Can someone confirm if this package was mistakenly imported?

My understanding of mosaic-streaming is that it takes a PySpark dataframe as input and provides API for getting a PyTorch dataloader as output which can be used for ML training on the fly without writing the whole data in some MDS format. Is my understanding correct?

PS: I started a discussion on your Slack community but was re-directed to submit an issue here.

Hi @gtmdotme, petastorm is not supported anymore as a DL dataloader on Databricks, my knowledge about pytastorm is also very limited :face_palm:
streaming is not available on earlier DBR versions, so you have to use 15.2 and above. On older DBR, you can manually install via pip install mosaicml-streaming==x.x.x
You understanding in pt-3 is correct. the tutorial is here
Let us know if you need more help.

Hi @XiaohanZhangCMU, thanks for your answers. I have a follow-up question on the third point. The tutorial you shared seems to indicate that we need to first convert parquet (pyspark dataframe) to MDS format. So, does that mean that we will consume double storage: one for stored parquet data (i.e., the PySpark Dataframe) and another for MDS format data?

@gtmdotme yes, that's correct. You will need to persist two copies.

We often store data in Parquet format (for PySpark dataframes), but to use your library, we need to also convert it to MDS format. This creates a second copy of the data, which could lead to a significant increase in disk storage usage. Is that a known tradeoff?
In my comparison for a synthetic dataset (example code below), I found that the MDS format is taking nearly 30x larger disk space than the Parquet format. I’m wondering if I'm doing something wrong or if this is expected behavior. This code is tested on the Databricks Runtime 15.4 LTS ML.
Thanks for your help!
Example
import shutil
import pandas as pd
from sklearn.datasets import make_classification
from streaming.base.converters import dataframe_to_mds

local_dir = './data_loading/'
shutil.rmtree(local_dir, ignore_errors=True)
os.makedirs(local_dir, exist_ok=True)

def get_dummy_data(n_samples, n_features, n_labels, seed):
    n_classes = n_labels  ## Max number of labels
    avg_labels_per_class = int(n_classes*0.1)  ## Number of averge labels per instance, here 10%

    # Generate dummy data
    X, y = make_multilabel_classification(n_samples=n_samples, n_features=n_features,
                                        n_classes=n_classes, n_labels=avg_labels_per_class,
                                        random_state=SEED)

    # Convert to pandas dataframe
    feature_cols = [f'feature_{i}' for i in range(n_features)]
    target_cols = [f'target_{i}' for i in range(n_classes)]
    X = pd.DataFrame(X, columns=feature_cols)
    y = pd.DataFrame(y, columns=target_cols)

    # Merge the group ids with the dataset
    data = pd.concat([X, y], axis=1)

    return data, feature_cols, target_cols

df, feature_cols, target_cols = get_dummy_data(n_samples=100_000, n_features=128, n_labels=32, seed=42)
print(df.shape)

# save as parquet, csv and numpy array
df.to_parquet(local_dir+'dummy_data.parquet')
df.to_csv(local_dir+'dummy_data.csv', index=False)
np.save(local_dir+"dummy_numpy.npy", df.to_numpy())

# convert pandas df to pyspark df
df_spark = spark.createDataFrame(df)
df_spark.write.mode("overwrite").save(local_dir+"dummy_spark")

# save the dataset to an MDS format
mds_kwargs = {'out': out_path}
dataframe_to_mds(df_spark.repartition(4), merge_index=True, mds_kwargs=mds_kwargs)

Output:
$ !du -achd1 {local_dir}

55M ./data_loading/dummy_data.csv
4.4M ./data_loading/dummy_data.parquet
123M ./data_loading/dummy_mosaic
123M ./data_loading/dummy_numpy.npy
5.2M ./data_loading/dummy_spark
310M ./data_loading/
310M total

Hey @gtmdotme Yes the trade-off of having an additional copy is known.
To help reduce the MDS copy size, you may want to add a compression method to mds_kwargs (for details, you can take a look at this page).
parquet compresses data efficiently if you have many repetitive values etc, so I am not surprised they lead to much smaller size since your MDS copy is just a serialized binary format.
Let me know what the size looks like after you applying the compression method.

Awesome! Compression did help. I used the zstd:9 and it drastically reduced the data size.
Here is the final code:
import os, shutil
import numpy as np
import pandas as pd
from sklearn.datasets import make_multilabel_classification
from streaming.base.converters import dataframe_to_mds
from pyspark.sql.functions import col

local_dir = './data_loading/'
shutil.rmtree(local_dir, ignore_errors=True)
os.makedirs(local_dir, exist_ok=True)

def get_dummy_data(n_samples, n_features, n_labels, seed):
    n_classes = n_labels  ## Max number of labels
    avg_labels_per_class = int(n_classes*0.1)  ## Number of average labels per instance, here 10%

    # Generate dummy data
    X, y = make_multilabel_classification(n_samples=n_samples, n_features=n_features,
                                        n_classes=n_classes, n_labels=avg_labels_per_class,
                                        random_state=42)

    # Convert to pandas dataframe
    feature_cols = [f'feature_{i}' for i in range(n_features)]
    target_cols = [f'target_{i}' for i in range(n_classes)]
    X = pd.DataFrame(X, columns=feature_cols)
    y = pd.DataFrame(y, columns=target_cols)

    # Merge the group ids with the dataset
    data = pd.concat([X, y], axis=1)

    return data, feature_cols, target_cols

df, feature_cols, target_cols = get_dummy_data(n_samples=100_000, n_features=128, n_labels=32, seed=42)
print(df.shape)

# save as parquet, csv and numpy array
df.to_parquet(local_dir+'dummy_data.parquet')
df.to_csv(local_dir+'dummy_data.csv', index=False)
np.save(local_dir+'dummy_numpy.npy', df.to_numpy())

# convert pandas df to pyspark df
df_spark = spark.createDataFrame(df)
df_spark.write.mode("overwrite").save(local_dir+"dummy_spark")

# save the dataset to an MDS format
out_path = os.path.join(local_dir, 'dummy_mosaic')
shutil.rmtree(out_path, ignore_errors=True)
mds_kwargs = {'out': out_path, 'compression': 'zstd:9'}
dataframe_to_mds(df_spark.repartition(4), merge_index=True, mds_kwargs=mds_kwargs)

Output:
!du -achd1 {local_dir}

55M     ./data_loading/dummy_data.csv
4.4M	./data_loading/dummy_data.parquet
6.1M	./data_loading/dummy_mosaic
123M	./data_loading/dummy_numpy.npy
5.2M	./data_loading/dummy_spark
193M	total

The DB Runtime 15.4 LTS ML comes with streaming version 0.7.4, however the latest is 0.9.0. I did upgrade to latest version because on the earlier version, I was running into the issue of converting array<int> datatype from PySpark to MDS format (which was fixed at a later release by you @XiaohanZhangCMU in this PR). Was upgrading to the latest version the right approach to fix this issue?
I have a column of type array<int> in the data and some of it's entries are empty array (i.e., []). While converting to MDS, it throws error:
ValueError: Attempting to encode a numpy array with 0 elements.

I tried to set those empty arrays to null but that also gives me error:
AttributeError: 'NoneType' object has no attribute 'dtype'

Is there a solution to this?

@gtmdotme Question to Q1 is yes. You will need to upgrade it which  has the array encoder.
For Q2, it's not a desired failure but sort of expected. MDS converter cannot do any imputation. It will expect the same dtype  for all records and they have valid values.
Q3: you are right, there is no boolean currently. I think a workaround is making it an integer and using "int".

Thanks so much for the clarification.
I think there are two limitations and subsequently improvements for a future release.

One, accommodating null values in the array datatype (which is a common scenario in ML).
Two, adding bool datatype in Mosaic ML's datatypes catalog (as using int instead of bool can take as much as 8x space).


Drop release-0.13 from dependabot config

It's obsolete.

Included with https://github.com/submariner-io/subctl/pull/1029


Change Total Interest Amount to Total Annual Interest on Borrow Loan screen for clarity

What UI/UX improvements would you like to be implemented:
To change label Total Interest Amount to Total Annual Interest.
Currently, Total Interest Amount can be misleading because users might think this is the immediate interest he/she has to pay, which can impact the amount of tokens he/she chooses to mint, especially in situations where a large loan amount is involved.

@ngyingzhong Just to confirm, are we going to change the wording here, too?

@ngyingzhong just to confirm, aren't we going to update the wording here, too?

@pierregee I don't think there's a need to change the wording in this page. In this context, it's quite evident that whatever amount you see on screen (e.g. 0.00023 dTSLA) is the amount of interest you have incurred. The reason I proposed the change for the Borrow Loan screen is because it's a loan overview page — in this "summary" view, I thought we should be careful with the details.

@ngyingzhong i was reviewing the PR fix for this then i realized currently the there is a bug that the Total annual interest is a flat amount. It does not change with the amount of loan that you plan to take. This is currently tracked in #1745
Aside from this, i am not sure if calling it total annual interest is a good idea, because the interest we used to display in Total loan + interest is the interest per block.
Interest per block = ( (vault interest + loan token interest) / 100 ) / (365 days * 2880 blocks in mainnet per year)
If we change to Total annual interest, we will need to display the value to be:
((vault interest + loan token interest) / 100 ) * (loan amount) ?

@kyleleow If total interest amount and total loan + interest is the interest for 1 block, not the entire year, it would invalidate my initial comments. My initial understanding was that it is reflecting the total annual interest amount, hence the feedback to change the text.

Let's calculate the annual interest on the front end, and display

Estimated annual interest
Total loan + annual interest


"Reserved parameters" are referred to as "response parameters"

Reserved parameters used to be referred to as response parameters previously, but now changed in the OIS format. The node/adapter code still refers to these as response parameters, even in error code 14.

Done. Just need to update error code 14

@acenolaza can this be closed now?

@acenolaza can this be closed now?

Yes, I believe we can. I'll also close 371


Does MoGenerator works fine with xCode 6.1 ?

I could not able to generate classes for newly added attributes after light weight migration
Any Bugs being on this front ?

If you're still having this problem, could you describe it in more detail? Nothing is obvious from your description.

Having heard anything. Closing it out.


Fix issue in validation method

Codecov Report

Merging #28 (8c0c3ce) into main (5843349) will increase coverage by 1.94%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##             main      #28      +/-   ##
==========================================
+ Coverage   87.50%   89.44%   +1.94%
==========================================
  Files           6        6
  Lines         352      341      -11
  Branches       58       55       -3
==========================================
- Hits          308      305       -3
+ Misses         31       27       -4
+ Partials       13        9       -4

Impacted Files
Coverage Δ

src/services/recommendations_service.py
86.95% <85.71%> (+4.50%)
:arrow_up:

src/repositories/recommendation_repository.py
100.00% <0.00%> (+4.08%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5843349...8c0c3ce. Read the comment docs.


Prevent Scalar Values in Update Messages

This is a better fix for the PR i submitted for the mongo storage layer crash:
https://github.com/deepstreamIO/deepstream.io-storage-mongodb/pull/7
In essence the message specification requires update message to contain an object, but it does not enforce it. Passing a null value through the the set, causes the mongo storage layer to crash. Presumably it would have a similar affect on other storage layers.
To prevent the round trip, this detect should be done within the client itself. I will submit a separate PR for the javascript client.

Hey, says you are a first time contributor, would you be okay signing our CLA?
https://deepstream.typeform.com/to/PwwNd5
Thanks!

The agreement mentioned a pdf version of the form that needs to be signed, but there was no link to that? Is that still required?

Nope, just signing it via the form should be enough. That was back before the digital age =P

I filled out the form.

Merging. Thank you for raising this!


reason of T5 download !?

why the script start downloading T5 while Flux contain it in repo !!!!  in the other hand , is there away to use civitai safetensores for training as usual  (single file), i think theres also away to train lora using unet alone

+1,same question.


Releasing Other Models?

First of all, thanks for the release! Second: its a pity that the released model (Resnet) has a highly non-standard layer ("Scale", which has a learnable bias parameter). That means that this model cannot be converted to other frameworks (Torch, MXNet, etc).
Would it be possible to release one of your other models as well, such as the Googlenet model?
Thanks!

WoW.......

Why did they only release the ResNet model and not the Googlenet model?

It works in TensorFlow converted with this tool.

No, the tool cannot convert it, exits with ValueError.

I used pycaffe to run the model successfully. However, used the cv::dnn (DNN module of opencv for loading caffe model) was failed. Is it because of the ResNet structure?

Unlock activet iphone 6s                                  IMEI<PHONE_NUMBER>23556

@zzzcpan install pycaffe first, ValueError is gone afterwards.

The Scale layer in Caffe is implemented in Batch Normalization layer in Torch. BN layer in Torch nn library has four parameters: weight, bias, running_mean and running_var, the weight and bias here are the factor and bias in Caffe Scale layer. So you could convert this model to Torch. Probably other platforms as well.
If you plan to convert this from Caffe to Torch, here are two things you might want to know:
In Caffe BN layer, there's a scale factor stored in eg. nsfw_net.params['bn_1'][2].data. You need to divide this factor from all mean and var before you copy these parameters to Torch BN layer.
The pooling layer default setting in Torch is different to Caffe. Mainly about padding, when the padding is not set up, the default value is 0 in both Torch and Caffe, but if the ( inputSize - kernelSize + 2 * padding) / stride is not an integer, in Torch the default setting is floor()(means no padding) while in Caffe it's ceil()(add one side padding each rather than two). A simple way to replicate this layer in Torch is nn.SpatialMaxPooling(kernelSize, kernelSize, stride, stride):ceil()
ref: https://netaz.blogspot.co.uk/2016/08/confused-about-caffes-pooling-layer.html?
Hope this is helpful.

I know I'm a bit late, but I've managed to convert the model to tensorflow ;) Hope it's useful for the one's having problems with the conversion process.
Check out the code and documentation at tensorflow-open_nsfw if you're interested.

@mdietrichstein Using this Tensorflowcode here: https://github.com/ProGamerGov/Protobuf-Dreamer, I was able to find DeepDream layer and channel combinations which produced hallucinations entirely comprised of specific categories: https://github.com/ProGamerGov/Protobuf-Dreamer/wiki/Interesting-Layers-And-Channels
It would be interesting to use the DeepDream "layer + channel" technique to see what exactly each category of the open_nsfw was. My experience with Tensorflow is very limited, as I mostly use Caffe and Torch7 at the moment.

@logic1988 I'm not sure what version of OpenCV you used but I'm using this model successfully with OpenCV DNN 3.3.1. You can see the use here.


[1.14.4] Crashing vanilla recipe book

MC 1.14.4 Forge 28.0.41 UselessMod-1.14.4-2.2.2
JEI does not work atm so I thought "vanilla has a recipe book, too". But when clicking the book the game crashes.
https://pastebin.com/25mHSBb9
 java.lang.AbstractMethodError: Method tk/themcbros/uselessmod/recipes/CompressorRecipe.func_194133_a(II)Z is abstract 	at tk.themcbros.uselessmod.recipes.CompressorRecipe.func_194133_a(CompressorRecipe.java) ~[?:1.14.4-2.2.2] {} 	at net.minecraft.client.gui.recipebook.RecipeList.func_194210_a(SourceFile:37) ~[?:?] {} ...

Jei Fixed btw, it is updated to forge .45

This issue will be fixed in the next version.
It crashes because of missing methods, which aren't needed in earlier versions of Forge
And JEI is updated. Use that. It's better anyway


How best to support patching objects with Schema Extensions

This SDK supports creating schema extensions, however it does not yet support for example adding an extensions data to an allowed object, for example a user.
Reading the documentation/examples it looks like we need to perform a patch call against the target object, however our update functions for users/groups etc requires a user/group object which has no concept of the schema extension (as in theory it could be anything).
I'd like to extend the SDK to support these kind of updates, however i'm looking to get some opinions on the best way to achieve this.
I can get myself around the problem by writing my own function that expects the schema extension i want to use, but i want to make this generic and add to the SDK.
Im still new to writing go so i'll welcome the most simple options  :-)

Capability merged in #81


Need a "f=" version of Minify_buildUri()

Pseudocode:

function Minify_pathsToUri($paths, $modRewriteWorks = false) {
  // $paths has strings like "//path/to/file.js"
  // converted into array of complete file paths
  // $mtime = max(filemtime(file1), filemtime(file2), ...)
  // construct URI
  $root = $modRewriteWorks ? "/min/f=" : "/min/?f=";
  return $root . implode(',', $shortPaths) . "&" . $mtime;
}

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 6 Aug 2009 at 12:05

there is a problem with this function

$root = $modRewriteWorks ? "/min/f=" : "/min/?f=";
return $root . implode(',', $shortPaths) . "&" . $mtime;

modRewriteWorks: false
http://examplem.com/min/?g=keyName&1111

modRewriteWorks: true
http://examplem.com/min/g=keyName&1111 the second url starts with & and ? is
missing. For me the browsers doesn't take the version number it should be
http://examplem.com/min/g=keyName?1111 because it was rewriten

Original comment by<EMAIL_ADDRESS>on 23 Apr 2014 at 5:56

What do you mean the "browser doesn't take the version number"?

Original comment by<EMAIL_ADDRESS>on 23 Apr 2014 at 12:27

Changed state: NeedInfo


Add ability to deploy your RStudio instance

What
When a user doesn't have this tool already deployed, a button to Deploy RStudio will be shown.
As part of these changes we now also show a link to the tool.
Why
So that users will be able to deploy RStudio without our intervention.
Environment variables
Before deploying we need to set the TOOLS_DOMAIN environment variable.
TODO

[x] Tests
[x] Possibly update the docker-compose.yaml

IMPORTANT: Depends on
Merge the following PRs before merging this.

[ ] https://github.com/ministryofjustice/analytics-platform-helm-charts/pull/63
[X] https://github.com/ministryofjustice/analytics-platform-config/pull/22

Ticket
https://trello.com/c/BbHaXQXt/562-2-cp-app-add-node-endpoint-to-deploy-a-tool

@axemonkey and @andyhd I don't know if you want to start to have a look at this.

Is this because the test times out at a shorter interval by default?

Correct.

(rebased)

I pushed some linting commits @andyhd

...also bear in mind this depends on https://github.com/ministryofjustice/analytics-platform-helm-charts/pull/63 @andyhd


fix: allow annotations to be added to the ServiceAccount

for integration with cloud workload identity on GKE/EKS
Signed-off-by: James Strachan<EMAIL_ADDRESS>

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign jstrachan
You can assign the PR to them by writing /assign @jstrachan in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/approve

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jstrachan
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [jstrachan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Add Frontend Routing

https://v5.reactrouter.com/web/guides/quick-start
Add Routes for landing page and home page tailored to users account.

closed by https://github.com/jduong09/gitcoding/pull/28


Binding site bb documentation ligand property

In the ligand property of the bindingsite bb, it says:
Ligand to be found in the protein structure. If no ligand provided, no action will be executed.
Actually, the binding site is searched and the bb is returning the residues, even if no ligand is provided.
Maybe needs rephrasing?
Thanks!

Modified by:
Ligand to be found in the protein structure. If no ligand provided, the largest one will be selected, if more than one.


Create variable code action not available for new expression

Description:
$subject.

One issue we are having related to the developer experience here is how many options are we gonna suggest. If we have multiple class definitions with the same init signature, then it is not scalable.

One issue we are having related to the developer experience here is how many options are we gonna suggest. If we have multiple class definitions with the same init signature, then it is not scalable.

This is not possible since we don't have any clue on what type will be returned or what really user want to be created. Create variable code actions only suggested on top of the assignment required diagnostic message. Thus closing this issue. Feel free to reopen if you need further clarifications.


Cowboy 1.0.4 uses Cowlib 1.0.2

Ran across the crypto:strong_rand_bytes/1 issue when compiling for 19.1. I see it is fixed in cowlib, but that fix is in a later build. Cowboy uses the default erlang.mk which looks like it is using Cowlib 1.0.2. Is that intentional? Does it require an upgrade to Cowboy 2.0 before Cowlib will invoke the new crypto call?
Looking at Cowboy 2.0.0-pre3 it invokes Cowlib 'master' rather than a tagged version.

I will tag Cowboy master and use tags for cowlib/etc. when releasing 2.0.0-pre.4 (almost done with the docs! so shouldn't be much longer).
For Cowboy 1.0.4 I don't remember exactly what the versions were but I made a mistake in Cowlib at some point and Cowboy 1.0.4 isn't compatible with it. From 2.0 onward Cowlib will be proper semver and this shouldn't happen anymore, but I'm afraid 1.0.4 won't be updated (customers not interested). The crypto thing should also just be a warning when Cowlib is used as a dependency so I don't think there's much problem there, is there?
2.0.0-pre.4 will also be the recommended version moving forward (in fact the current master code is it, just missing some docs).

It's now deprecated as a warning, but I was compiling with warnings as errors and that was the only error that tripped me. I can wait for 2.0. Still doing an initial deploy so I can make the first release with the latest, greatest code during December.

Right. Wish I finished with the new version sooner, but customers have been very demanding recently. Hopefully can finish soon.

This still isn't fixed yet. Now getting: Warning: call to crypto:rand_bytes/1 will fail, since it was removed in 20.0; use crypto:strong_rand_bytes/1

There are no Cowboy 1.x versions planned. Update to 2.0!


64-bit crosswalk works?

I went ahead and made 64-bit versions of my application and just assumed crosswalk worked with those architectures.  I just saw a post on another issue that says it doesn't work.  Unfortunately, I don't have a 64-bit device to actually test it.  Has anyone confirmed one way or the other whether crosswalk works in x86_64 or arm64?  The size of my apk files implies binaries are included.

Thanks for the info @JacobSchwartzSoerensen! I have created two versions of a test app: One 32 bit arm version and one 64 bit one. I can see the following folder structure (only showing you the lib folder)
32 bit
armeabi-v7a/
armeabi-v7a/libxwalkcore.so (~36 mb)
armeabi-v7a/libxwalkdummy.so (~6 kb)

64 bit
arm64-v8a/
arm64-v8a/libxwalkcore.so (~58 mb)
arm64-v8a/libxwalkdummy.so (~6 kb)

Are there any more libraries/files that I should look for? Or is this a sign that 64 bit works correctly?

I also see xwalk.pak and xwalk_100_percent.pak in the res/raw/ folder in my working APK's, but other than that, it looks good.

I am closing this issue since there was no activity within the last 4 weeks. If you have any questions, feel free to comment on this issue or to create a new one.

Thanks for the info @JacobSchwartzSoerensen! I have created two versions of a test app: One 32 bit arm version and one 64 bit one. I can see the following folder structure (only showing you the lib folder)
32 bit
armeabi-v7a/
armeabi-v7a/libxwalkcore.so (~36 mb)
armeabi-v7a/libxwalkdummy.so (~6 kb)

64 bit
arm64-v8a/
arm64-v8a/libxwalkcore.so (~58 mb)
arm64-v8a/libxwalkdummy.so (~6 kb)

Are there any more libraries/files that I should look for? Or is this a sign that 64 bit works correctly?

What changes did you make in your code that created the new folder arm64-v8a?

I don't remember the details of this anymore, but have a look at the comments in the referenced issue
Maybe try building with --xwalk64bit

@shishirshettygit It should be possible to create a 64 bit build using the changes in config.xml as described in the README of this repo.

Adding

will only generate arm64 apk. No x86_64. More, the best would be generating only 1 apk with all 4 platforms inside. is it possible?

@xale76
See my comment in #3


How can I Call an intent from LUIS from session

SDK Platform
 Node SDK
URL of Code Sample

Problem Description

When user connect to the bot, user want to make some intent request, but  before my bot can help users, the bot need to check their credentials, for example, username and password. If users havent logined to the system, the bot requires users login and after-that forward to the intent requests of users.
Now I am finding the way how to forward intents from session instead of getting them from user prompt.
Please share me some experiences with this problem.
Thanks.

@binhpht did you manage to make a intent call when starting your bot? I´m looking for a similar approach, my bot would start a intent call to LUIS before redirecting into a specific Dialog.
Regards

The sdk and framework now provide authentication support.  Some documentation: https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-authentication?view=azure-bot-service-4.0&tabs=javascript  and sample: https://github.com/Microsoft/BotBuilder-Samples/tree/master/samples/javascript_nodejs/18.bot-authentication
After a successful login, messages can be processed by LUIS.


[Week 9] Rename AB3/AddressBook to ConText

Renamed all naming/reference to AB3/AddressBook to ConText. I left out namings in the .puml files. These are files associated with diagrams like class diagrams and stuff. I am not sure whther they will be relevant in the future. We can make changes on them when we touch on that part.
Closes #4, closes #41.

Just some initial thoughts, open for discussion.
Also, should we take the opportunity to rename the packages early?

Do i just rename the package altogether in this PR or separately?

As Bryan has pointed out, I also fixed the link to our UserGuide. From what I observe, I think it is the only place to change. If I did not miss anything, I think we can close #41 as well with this PR :)

I would like to discuss the naming. Perhaps the original work item wasn't very clear.
Old

AddressBook implements ReadOnlyAddressBook, containing persons
JsonAddressBookStorage implements AddressBookStorage , handling the JSON side of storing ReadOnlyAddressBook via JsonSerializableAddressBook

StorageManager implements Storage, handling AB3's storage via both AddressBookStorage and UserPrefsStorage

AddressBookParser handles AB3's parsing by using individual command parsers

Current PR

ConText implements ReadOnlyConText, containing contacts
JsonConTextStorage implements ConTextStorage , handling the JSON side of storing ReadOnlyConText via JsonSerializableConText

StorageManager implements Storage, handling ConText's storage via both ConTextStorage and UserPrefsStorage

ConTextParser handles ConText's parsing by using individual command parsers

Proposed?

ContactsManager implements Contacts, containing contacts
JsonContactsStorage implements ContactsStorage , handling the JSON side of storing Contacts via JsonSerializableContacts

StorageManager implements Storage, handling ConText's storage via both ContactsStorage and UserPrefsStorage

InputParser handles ConText's parsing by using individual command parsers

I find it hard to express exactly why, but I feel we shouldn't use ConText as is in the class names. Though context would fit in the package name.
One reason pointed out above was how ConText refers to the app as a whole, but the current class named after it handles a subset of the app's storage - just the Contacts - as part of the overall storage component. The name on its own is a little uninformative in that it doesn't really explain what the class is for. This was not a problem for AB3 as AddressBook is more self-explanatory, so it could overlap with the cosmetic naming.
The capitalisation style of our name also makes it feel a bit clunky to use in our code, eg when trying to camel case it (see below).

Other things noted:

Class was renamed to JsonSerializableConText but file is still JsonSerializableAddressBook.java
At least one instance of variable name not being camel case: public StorageManager(ConTextStorage ConTextStorage, ..., where it should be conTextStorage

I do agree with you on the renaming the class files to context. I was reading through earlier and felt a bit weird seeing everything be named ConText. It doesn't feel very friendly from a new developer point of view. Maybe it would be better to rename AddressBook references as ContactList instead.

I worry if a ContactList that contains the existing UniqueContactList becomes ambiguous.
I also read this SO answer, specifically its second bullet point, which made me wonder if the ReadOnly part of the interface names are too long compared to the shorter class names. So instead it came to mind to adopt some of the existing naming scheme where some interfaces' implementations have Manager appended.

I would like to discuss the naming. Perhaps the original work item wasn't very clear.
Old

AddressBook implements ReadOnlyAddressBook, containing persons

JsonAddressBookStorage implements AddressBookStorage , handling the JSON side of storing ReadOnlyAddressBook via JsonSerializableAddressBook

StorageManager implements Storage, handling AB3's storage via both AddressBookStorage and UserPrefsStorage

AddressBookParser handles AB3's parsing by using individual command parsers

Current PR

ConText implements ReadOnlyConText, containing contacts

JsonConTextStorage implements ConTextStorage , handling the JSON side of storing ReadOnlyConText via JsonSerializableConText

StorageManager implements Storage, handling ConText's storage via both ConTextStorage and UserPrefsStorage

ConTextParser handles ConText's parsing by using individual command parsers

Proposed?

ContactsManager implements Contacts, containing contacts

JsonContactsStorage implements ContactsStorage , handling the JSON side of storing Contacts via JsonSerializableContacts

StorageManager implements Storage, handling ConText's storage via both ContactsStorage and UserPrefsStorage

InputParser handles ConText's parsing by using individual command parsers

I find it hard to express exactly why, but I feel we shouldn't use ConText as is in the class names. Though context would fit in the package name.
One reason pointed out above was how ConText refers to the app as a whole, but the current class named after it handles a subset of the app's storage - just the Contacts - as part of the overall storage component. The name on its own is a little uninformative in that it doesn't really explain what the class is for. This was not a problem for AB3 as AddressBook is more self-explanatory, so it could overlap with the cosmetic naming.
The capitalisation style of our name also makes it feel a bit clunky to use in our code, eg when trying to camel case it (see below).
Other things noted:

Class was renamed to JsonSerializableConText but file is still JsonSerializableAddressBook.java
At least one instance of variable name not being camel case: public StorageManager(ConTextStorage ConTextStorage, ..., where it should be conTextStorage

Just fixed issues listed in Other things noted

Welcome to Codecov :tada:
Once merged to your default branch, Codecov will compare your coverage reports and display the results in this comment.
Thanks for integrating Codecov - We've got you covered :open_umbrella:

Very strange bug indeed


Added Docs and Examples!

Ngrx is an amazing Reactive-Redux implementation for Angular 2. Ngrx/db is an important tool our toolbag for preserving the state when the page is closed or refreshed. This repo has gotten NO love at all in terms of the README, and a lot of people have been / still are very confused about it. Imo we should all just start using it and work out the bugs (if there are any) when people open issues, just like any other project.
Anyway, I'm open to feedback and criticism if anything I wrote is totally wrong. Just let me know or change it.
Thanks,
Jimbo

ok. 👍
I just changed to my branch to have the all lowercase capitalization ngrx/db.

I'm just watching this, but .idea folder is necessary? Can/Should it be removed?

Ah sorry. I didn't realize I added that. It's an artifact created by WebStorm / intellij. You can delete it. :+1

add a rule in .gitignore will avoid it.

ok. I just deleted the idea folder so it's no longer included in this PR. :)

Hey, how is it looking now? Can we merge this? :)

Please guys, the entire world hates us for this embarrassing state of the current readme.

Can someone merge this?

I'm not a collaborator or anything, but this seems more like an internal project, that something that someone else could really use.
It looks really good, but I don't think this is the right time to change the readme yet.

@michaeljota
I'm assuming you mean "not something that someone else could really use." ?
And if so, why on earth would you think that???

You may want something, yet be unable to use it. As this.

What evidence do you have that this is unable to be used? I am using it right now, as are many other devs I know:

https://github.com/xlozinguez/angular-nyc-advanced-ngrx
https://github.com/btroncone/ngrx-examples
https://github.com/dereklin/ng-ws-ngrx-hmr
https://github.com/JimTheMan/t2

The fact that someone is using it, it's not proof to say it should be used.

Well we are all using it, and it's working fine. I'm even using in production for one app...
What would be sufficient proof that it should be used?
BTW anyone who actually wrote this library (@mikeryan52 @brandonroberts @robwormald) feel free to chime in...

Nope. I mean, the people who actually wrote this say "don't use me yet".

 ;)
On Mar 3, 2017 7:51 PM, "Michael De Abreu"<EMAIL_ADDRESS>wrote:

Nope. I mean, the people who actually wrote this say "don't use me yet".
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ngrx/db/pull/16#issuecomment-284110707, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AFGysxplirKm9FPCWKfFINccu8MF9K6Jks5riLWCgaJpZM4MGfv-
.

🙄🙄🙄🙄 You must be the rudest person I ever chat here in Github. They have no obligation to even have their own repo documented in anyhow. Thanks you for your README, if that's what you want, but I hope this gets closed, along with your duplicate issue #16 :).

@michaeljota Please stop trolling this thread.

Agreed, @michaeljota your personal attacks don't add any value to this or any project. I think @JimTheMan  should be commended for atleast trying to contribute something. And the open source maintainers do have some level of obligation, IMO to atleast respond "we have seen your comments/PR, but we are busy at the moment and will respond at such and such a time"
Good work @JimTheMan

@somombo Just for clarification, I'm not personal attacking him. I'm watching this whole repo, so I can track progress. If you look in issues, you could see an open issue that only refers to merge this, still open, PR.
So I'm just leading to the fact that the people behind this project may not want to make this public usable yet, and this guy keep pushing to merge something as it is some kind of obligation for this organization to merge every PR.
I just want a notification here to be related with progress. I'm sorry if there is any misunderstanding.

@JimTheMan's opinions are valid. A coordinated effort can make it usable.

@JimTheMan thanks for the PR. i'm merging it now and then i'll be migrating this repo to the monorepo @ https://github.com/ngrx/platform
to everyone commenting in here:
ngrx/db began and remains an experimental thing. IndexedDB is a non-trivial API, and notoriously difficult to use consistently across different browsers. I'm not comfortable committing to the full support we give the other ngrx projects until it's been much more extensively tested and proven, which is why it has not previously had docs or a readme.
I, frankly, don't have the hours in the day to make that happen right now. If the community would like to make that happen, I am all for it and would welcome your help.


lug: another try of mirrorz adhoc support

#440 again, with compatibility to lug's Config struct this time.

LGTM, let's give it another shot.


Update quick-setup-auto-provision.md

iothubownert to IoT hub Owner

@shrazaghi I still see "iothubowner" as the name of the access policy in the portal. Do you see something different?

@shrazaghi I'm going to close this pull request. If you disagree with this decision, please let me know why and we can reopen the PR. Thank you.
#please-close


messages: remove circular imports in typescript code, add linting

Details
Remove circular imports in messages TypeScript code and add eslint-plugin-import to lint against future regressions.
Motivation and Context
See #1279 which this helps with but doesn't close, because we have a similar issue in the protobufjs library.
These issues block us upgrading to webpack 5 in the HTML formatter. Node's module resolution handles them okay and apparently so does webpack 4, but the runtime module resolution seems to have changed somehow with 5.
Types of changes

[x] Bug fix (non-breaking change which fixes an issue).
[ ] New feature (non-breaking change which adds functionality).
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected).

Checklist:

[ ] The change has been ported to Java.
[ ] The change has been ported to Ruby.
[ ] The change has been ported to JavaScript.
[ ] The change has been ported to Go.
[ ] The change has been ported to .NET.
[x] I've added tests for my code.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have updated the CHANGELOG accordingly.

@aslakhellesoy thanks for moving this forward, I hadn't caught on that it would fail unless all eslint configs were the same (this should get easier if we can make npm workspaces happen). I see there are some failures on this rule in gherkin and cucumber-expressions, shall I pick this back up and work on those? (It's not super urgent to fix this, just prevents upgrading to webpack 5.)

Hi @davidjgoss yes, that would be great if you could continue the untangling of the circular deps in gherkin and cucumber-expressions.
Thanks!!

Allright, that's cucumber-expressions done. I'll have a quick go at gherkin too....

Gherkin circular dependency:

I think Gherkin circulars are gone now - still some small things to fix in cucumber-expressions it seems...

You opened a can of worms here @davidjgoss :-)
But I'm glad you did. Keeping code nicely decoupled is something I care deeply about, and this has been an eye opener for me. It's the first time (I think) that I have worked with a system that disallows circular dependencies at the compilation level.
I've not seen it in Java or C# before, or any other language (although I'm sure it exists).
In order to untangle these circular dependencies I was forced to do what I have considered good practice for 15 years - interface-implementation separation. But I don't always practice what I preach, so having a tool that enforces this is great!
I'll be using this on all my TypeScript code from now on I think.
@mpkorstanje is there anything similar in Java that you know of?

@aslakhellesoy awesome, thanks for getting it over the line (i.e. the last, er, 99%)


Design configurability and dependency management for modular serverless

Description
Changes proposed in this pull request:

name serverless dependencies
design main configurability capabilities of the Serverless CR and its status
describe configuration propagation

Related issue(s)
https://github.com/kyma-project/serverless-manager/issues/3

Looks very good, thank you. Just a couple of nit comments.


Use envoy_cc_library instead of cc_library for apple_platform_cert_verifier

Use envoy_cc_library instead of cc_library for apple_platform_cert_verifier
Risk Level: N/A
Testing: N/A
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features: N/A

/assign @Augustyniak

/retest


Added various endpoints for subscriptions

Added endpoints for retrieving subscriptions to:

Forms

https://developers.convertkit.com/#list-subscriptions-to-a-form

Sequences

https://developers.convertkit.com/#list-subscriptions-to-a-sequence

Tags

https://developers.convertkit.com/#list-subscriptions-to-a-tag

@ericalli the pull request failed, if you make it pass, I might be able to merge it

@myKingEdem is there any further work that is required for this PR to be merged?


improve issue template.

As #3389 , I created issue templates. You can check how it works in my repository.
I created the templates from GitHub default templates and babel. I'm not native English. So, feel free to point out wrong grammar or better expressions.

Coverage increased (+0.05%) to 90.063% when pulling ed23b399be43db90e8bd2e395e3aa25f76e79182 on outsideris:issue-3389 into eeccd053b18361c342dff02d4f728bd439c7f277 on mochajs:master.

I think GitHub supports auto label now.

Optionally, to automatically set a default issue title or add labels and assignees to your issue template, add a YAML frontmatter to your issue template with title, labels, or assignees.

I will update them.

Done.

Done.

Fixed.


A big THANK YOU

Hi @receter I would like to thank you for the repository and the guide at dev.to.
I have a task to create a component library with Mantine + Wagmi. This is exactly what I need.
Again, thank you very much!


[Documentation] Refactor: Reorganize Connect sidebar and simplify guides

Problem solved
Short description of the bug fixed or feature added

PR-Codex overview
This PR focuses on updating the content of various documentation pages related to wallet integration, specifically distinguishing between in-app wallets and ecosystem wallets. It also refines sidebar navigation and improves organization.
Detailed summary

Updated descriptions for ArticleIconCard components to reflect "in-app wallets" vs. "ecosystem wallets."
Adjusted sidebar links for better organization and clarity.
Removed outdated or redundant links and sections.
Enhanced consistency in terminology across documentation.

✨ Ask PR-Codex anything about this PR by commenting with /codex {your question}

#5625  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.


Add Quark type

Removed the two functions.

@GuillaumeGomez Please remove generated intern_(static_)?string too

@EPashkin They're already removed no? Or are you talking about something outside of Quark?

@GuillaumeGomez I talking about https://github.com/gtk-rs/glib/blob/master/src/auto/functions.rs#L614

@EPashkin If I remove them, we won't have access to the intern string anywhere. If it's what you want then I'll just remove them.

That API makes no sense from Rust, you can remove :)

@GuillaumeGomez Yes, I think that both function unusable in rust

I did. I think it's a first for me to remove things instead of adding ones haha.

Updated.

Perfect, I'm offline again :)

👍

Thanks to both of you for your reviews! :)


set enable_ecs_service_role to true only if there is a load balancer

When creating service, and there is no LB defined "You cannot specify an IAM role for services that require a service linked role."
Set enable_ecs_service_role to false when there is 0 LB or more than 1.
what
When creating service with network mode != awsvpc and there is no LB defined it throws error: "You cannot specify an IAM role for services that require a service linked.
why
When setting value enable_ecs_service_role we check if there is no more than 1 LB, but less than 1 is 0 and we can't assign IAM role when it is no LB and network mode is bridge, host or none.
solution
Set enable_ecs_service_role to false when there is 0 LB or more than 1 when network mode is not awsvpc

/test all

Blocked by https://github.com/cloudposse/terraform-aws-security-group/pull/17

@nitrocode this PR (#118) should be closed and this one re-opened:
https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/pull/137
The code change in this PR (#118) is wrong.

Sorry for inactivity, added a fix commit as is in pull request #137. Thank you for help

@n0rig just FYI, we are still waiting on some related fixes to happen before merging this one


Temporarily disable vulnerability scanning

This starts utilizing a new token that has access to write to discussions

@philipdouble @sdstolworthy I think a little more research is required here on exactly what I can do with tokens -- so just going to disable this for now. Could one of you approve?


pkg/rancher-components/StringList: add more unit tests

Relates to rancher/dashboard#7326

I'm adding dev-validate to this ticket - it's adding unit tests. nothing for qa to test on this one.


XSS possible in articles via javascript: links

I think there should be a whitelist of protocols:
[click me](javascript:alert(document.cookie))

Hmm, that seems like a vulnerability in Redcarpet.


TLS connection error with OTP 25

TDS connection via TLS fails with OTP 25 with following error:
[error] Tds.Protocol (#PID<0.621.0>) failed to connect: ** (Tds.Error) Unable to establish secure connection to server due :closed
[error] GenServer #PID<0.637.0> terminating
** (stop) :tcp_closed
Last message: {:tcp_closed, #Port<0.22>}
State: %Tds.Tls{buffer: nil, handshake?: true, owner_pid: #PID<0.664.0>, socket: #Port<0.22>, ssl_opts: [active: false, cb_info: {Tds.Tls, :tcp, :tcp_closed, :tcp_error}]}
[warning] Description: 'Authenticity is not established by certificate path validation'
     Reason: 'Option {verify, verify_peer} and cacertfile/cacerts is missing'

[error] GenServer #PID<0.629.0> terminating
** (stop) :tcp_closed
Last message: {:tcp_closed, #Port<0.27>}
State: %Tds.Tls{buffer: nil, handshake?: true, owner_pid: #PID<0.661.0>, socket: #Port<0.27>, ssl_opts: [active: false, cb_info: {Tds.Tls, :tcp, :tcp_closed, :tcp_error}]}
[error] GenServer #PID<0.636.0> terminating
** (stop) :tcp_closed
Last message: {:tcp_closed, #Port<0.25>}
State: %Tds.Tls{buffer: nil, handshake?: true, owner_pid: #PID<0.659.0>, socket: #Port<0.25>, ssl_opts: [active: false, cb_info: {Tds.Tls, :tcp, :tcp_closed, :tcp_error}]}
[error] Tds.Protocol (#PID<0.617.0>) failed to connect: ** (Tds.Error) Unable to establish secure connection to server due :closed
[error] Tds.Protocol (#PID<0.624.0>) failed to connect: ** (Tds.Error) Unable to establish secure connection to server due :closed
[error] GenServer #PID<0.628.0> terminating
** (stop) :tcp_closed
Last message: {:tcp_closed, #Port<0.20>}
State: %Tds.Tls{buffer: nil, handshake?: true, owner_pid: #PID<0.665.0>, socket: #Port<0.20>, ssl_opts: [active: false, cb_info: {Tds.Tls, :tcp, :tcp_closed, :tcp_error}]}
[error] Tds.Protocol (#PID<0.618.0>) failed to connect: ** (Tds.Error) Unable to establish secure connection to server due :closed

No additional ssl_opts are set.

@bosko
There was a new release of OTP just now (https://github.com/erlang/otp/releases/tag/OTP-25.0.1).
It contains some fixes regarding SSL handling.
Could you try 25.0.1 to check if this resolves the problem?

@moogle19
It does resolve the problem. I've checked both TDS versions 2.3.0 and master branch and both work.


NETOBSERV-1898 & NETOBSERV-1896: refactor pkt drop to be more efficient with using stack [BP 1.7]

Signed-off-by: Mohamed Mahmoud<EMAIL_ADDRESS>(cherry picked from commit 48d8fb3309703cb017b13188c0116b681b320c28)
Description

Dependencies

n/a
Checklist
If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.

[ ] Will this change affect NetObserv / Network Observability operator? If not, you can ignore the rest of this checklist.
[ ] Is this PR backed with a JIRA ticket? If so, make sure it is written as a title prefix (in general, PRs affecting the NetObserv/Network Observability product should be backed with a JIRA ticket - especially if they bring user facing changes).
[ ] Does this PR require product documentation?

[ ] If so, make sure the JIRA epic is labelled with "documentation" and provides a description relevant for doc writers, such as use cases or scenarios. Any required step to activate or configure the feature should be documented there, such as new CRD knobs.

[ ] Does this PR require a product release notes entry?

[ ] If so, fill in "Release Note Text" in the JIRA.

[ ] Is there anything else the QE team should know before testing? E.g: configuration changes, environment setup, etc.

[ ] If so, make sure it is described in the JIRA ticket.

QE requirements (check 1 from the list):

[ ] Standard QE validation, with pre-merge tests unless stated otherwise.
[ ] Regression tests only (e.g. refactoring with no user-facing change).
[ ] No QE (e.g. trivial change with high reviewer's confidence, or per agreement with the QE team).

/approve

/test all

/retest

/ok-to-test


Italicize Function

Not sure if it has been tried yet, but a way to italicize some of the outputted text in the SRS. For example, in any reference section, the title of the reference is italicized. Something that could be implemented the way Quote()  or sParen() are implemented.

What we really need is some semantic markup (title in this case) which, when coupled with a 'style' file, can render titles in italics.  It's like XML+CSS.  We don't want to have italics hard-coded in.  We want to move away from display-based encoding of knowledge towards more semantics-based encoding coupled with 'rendering' instructions that are display-based.

@danscime can you point out a few specific examples in the text that you are referring to? Depending on where these are showing up they may have several different "meanings" that need to be explored.

So far, I've only seen it in the reference section as well as variable symbols throught the SWHS SRS.
I guess it's a way to know that it is a symbol so as not to be confused with regular text

@danscime The italicizing of symbols is taken care of in LaTex currently with dollar signs "$V_W$". HTML should use some math render then it too can have italicized symbols.

@danscime Italicizing references will be handled automatically once #245 is done, and as @njericha mentioned, italicizing is automatically handled by the generator (in the case of TeX).
For the HTML, symbol emphasis is missing, so that is definitely something that will need to be fixed before we close this issue.

The way this was implemented in the HTML backend is rather sub-optimal.  I'll reopen this (and assign it to myself), to remember to fix it.


Is this repo still maintained?

Is this still the best way to ship logs from Kinesis to Sumologic?
If not, could you please point me to the better way?
If so, are you open to PRs to bring things up to date and/or address issues reported in the repo?

We do monitor this. Are you collecting your own logs via Kinesis or from other AWS services?

We are deciding the best way to collect a significant flow of logs.  Kinesis is attractive because it has such high throughput capabilities.  Sumologic links to this repo for shipping logs from Kinesis.
With the most recent commits being ~3 years ago, I just wanted to confirm that this is still the best solution before putting work into setting it up.

@duchatran I've opened a PR that corrects the Ant dependencies so that the app will start up using the commands in the README.  I am working to address a couple other issues reported in the repo that I also find necessary, but don't want to flood the repo with PRs all at once.
If you have time, I would greatly appreciate a review/merging of the PR.

Done, thanks for contributing @truppert-mdsol. Feel free to submit PR to update README as well. Closing this issue for now.


WebView ERR_ACCESS_DENIED for local files on Android SDK 30

Steps to Reproduce
Consider the following code
import 'dart:io';

import 'package:flutter/material.dart';
import 'package:path_provider/path_provider.dart';
import 'package:webview_flutter/webview_flutter.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatefulWidget {
  // This widget is the root of your application.
  @override
  _MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
  String file;

  @override
  void initState() {
    ()async{
      var basePath = (await getApplicationSupportDirectory()).path;

      file = "$basePath/1.htm";

      File(file).writeAsString("
Test page contents
");
      setState(() {});
    }();

  }

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      home: file == null?Container():WebView(
        initialUrl: Uri.file(file).toString(),
      )
    );
  }
}

with dependencies
  webview_flutter: ^1.0.7
  path_provider: ^1.6.22

Run it on android 30 emulator/device.
If you have in build.gradle
compileSdkVersion 29
targetSdkVersion 29

everything works as expected (you see WebView with test contents).
But it if you change to
compileSdkVersion 30
targetSdkVersion 30

WebView shows net::ERR_ACCESS_DENIED
It looks like this issue comes from changes of default value of https://developer.android.com/reference/android/webkit/WebSettings#setAllowFileAccess(boolean)
We can try to allow changing this setting from flutter widget (which is looks like not good, because setAllowFileAccessFromFileURLs is deprecated).
And/or, maybe better, to implement the recommended mechanism of intercepting url loading (WebViewAssetLoader). In this case, it will be possible to implement file loading from custom scheme in the own code.

[√] Flutter (Channel stable, 1.22.4, on Microsoft Windows [Version 10.0.19042.662], locale uk-UA)
    • Flutter version 1.22.4 at D:\Programs\Android\flutter
    • Framework revision 1aafb3a8b9 (3 weeks ago), 2020-11-13 09:59:28 -0800
    • Engine revision 2c956a31c0
    • Dart version 2.10.4

[!] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at D:\Programs\Android\sdk
    • Platform android-30, build-tools 30.0.2
    • ANDROID_SDK_ROOT = D:\Programs\Android\sdk
    • Java binary at: D:\Program Files\Android\Android Studio\jre\bin\java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b01)

[√] Connected device (1 available)
    • sdk gphone x86 arm (mobile) • emulator-5554 • android-x86 • Android 11 (API 30) (emulator)

@ertong
Add below permission in manifest and try again to see if it works:
<uses-permission android:name="android.permission.QUERY_ALL_PACKAGES"/>

@darshankawar
Thanks for suggestion.
In my setup, result is the same SDK 29 - works, SDK 30 - access denied.

Replicable when sdk is set to 30. It shows:

With sdk set to 29, data shows up properly:

flutter doctor -v
[✓] Flutter (Channel stable, 1.22.4, on Mac OS X 10.15.4 19E2269 darwin-x64,
    locale en-IN)
    • Flutter version 1.22.4 at /Users/dhs/documents/Fluttersdk/flutter
    • Framework revision 1aafb3a8b9 (3 weeks ago), 2020-11-13 09:59:28 -0800
    • Engine revision 2c956a31c0
    • Dart version 2.10.4

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.0)
    • Android SDK at /Users/dhs/Library/Android/sdk
    • Platform android-30, build-tools 30.0.0
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.9.3

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 46.0.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.51.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.16.0

[✓] Connected device (2 available)
    • sdk gphone x86 arm (mobile)         • emulator-5554
      • android-x86 • Android 11 (API 30) (emulator)
    • iPhone SE (2nd generation) (mobile) • 6C85835D-FBFD-4AB3-8DE8-B4FAD35E5367
      • ios         • com.apple.CoreSimulator.SimRuntime.iOS-14-0 (simulator)

• No issues found!

Probably has to do with giving access to load local files / html with sdk 30.
May be similar to https://github.com/flutter/flutter/issues/27086

Hey, I have the same issue, I can't access any files from documentDirectory inside webview in SDK 30.
Any solution?

Adding webView.getSettings().setAllowFileAccess(true); in FlutterWebView.java works, as suggested by @ertong. The method is also not deprecated I think, the method below it in the docs (setAllowFileAccessFromFileURLs) is.
I guess we should make this configurable via Flutter as the Android docs mention security reasons as reasons why the default was set to false.

It works for me!
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

android:usesCleartextTraffic="true"

webSettings.setAllowFileAccess(true);
webSettings.setAllowFileAccessFromFileURLs(true);

It works for me!
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

android:usesCleartextTraffic="true"

2980×1068
webSettings.setAllowFileAccess(true);
webSettings.setAllowFileAccessFromFileURLs(true);

Are all of those necessary for you? For me it works with just webSettings.setAllowFileAccess(true);

Where can I change the file webviewbuilder.java?
Thanks

While the webView.getSettings().setAllowFileAccess(true); fix works for v1.x.x of the webview_flutter package, I can't seem to get it working on v2.1.1 (latest). Since the structure of FlutterWebView.java changed, I attempted to add webView.getSettings().setAllowFileAccess(true); into the FlutterWebView constructor but no luck.
@Aulig based on your code snippet, I'm assuming you were modifying a previous version of the package? Has anybody been able to get this working on a modern version of the package and if so, what code was modified?

Hi @alexagat - you're correct, I modified an old version. I didn't notice that it changed in newer versions. I'd assume that you could just add it to the webviewbuilder in the createWebView function something like this:
webViewBuilder
        .setUsesHybridComposition(usesHybridComposition)
        .setDomStorageEnabled(true) // Always enable DOM storage API.
        .setJavaScriptCanOpenWindowsAutomatically(
            true) // Always allow automatically opening of windows.
        .setSupportMultipleWindows(true) // Always support multiple windows.
        .setWebChromeClient(webChromeClient)
        .setAllowFileAccess(true) // <-- HERE'S MY CHANGE
        .setDownloadListener(
            downloadListener); // Always use {@link FlutterWebChromeClient} as web Chrome client.

But I haven't tested that. Or is that what you already tried Alex?

But I haven't tested that. Or is that what you already tried Alex?

@Aulig correct, I tried a number of variations including adding the .setAllowFileAccess(true) as a chained call to the webViewBuilder (as your snippet shows). If I downgrade back to 1.0.7 (the version I was at before trying to upgrade to 2.1.1) the fix works.
I'll have to bump the version 1 by 1 to try and identify the specific bump of the webview_flutter package that broke this fix unless anybody else has ideas..

I see, that's unfortunate. I think I forked off version 2.0.2, which works for me. So you can start your search there if you'd like. Good luck!

webSettings.setAllowFileAccess(true);
webSettings.setAllowFileAccessFromFileURLs(true);

Thank you is worked

Same issue here

None of the above solutions are working since the plugin was updated rapidly in the few months and I think the structure had  big changes and modifications

P5 is way to low for this

wokaround for android 10 webview_flutter_android 2.8.0
go to file /flutter/common/flutter/.pub-cache/hosted/pub.dartlang.org/webview_flutter_android-2.8.0/android/src/main/java/io/flutter/plugins/webviewflutter/WebViewHostApiImpl.java
and make sure that create method has setAllowFileAccess and setAllowFileAccessFromFileURLs
example
`  @Override
public void create(Long instanceId, Boolean useHybridComposition) {
DisplayListenerProxy displayListenerProxy = new DisplayListenerProxy();
DisplayManager displayManager =
(DisplayManager) context.getSystemService(Context.DISPLAY_SERVICE);
displayListenerProxy.onPreWebViewInitialization(displayManager);
final WebView webView =
    useHybridComposition
        ? webViewProxy.createWebView(context)
        : webViewProxy.createInputAwareWebView(context, containerView);

 webView.getSettings().setAllowFileAccess(true); //---------------------------add this
 webView.getSettings().setAllowFileAccessFromFileURLs(true);//------------------and this

displayListenerProxy.onPostWebViewInitialization(displayManager);
instanceManager.addInstance(webView, instanceId);

}`

adding the line
webView.settings.setAllowFileAccess(true);
under the class WebViewAndroidPlatformController along with the other settings in the file webview_android_widget.dart in the dependency plugin webview_flutter_android  works, local files can be accessed
checked with version webview_flutter_android 2.8.2 & android sdk 31

@vgsrivathsan approach also works for me. Will there be an update in the webview plugin to be able to enable this? Currently I have to add the plugin as a local dependency to my Code and add the line from @vgsrivathsan to it.

Looks like the issue is resolved when using WebView's upstream loadFile method: https://pub.dev./packages/webview_flutter/changelog#240
This works for me - tested with some files located below getTemporaryDirectory() of path_provider package.

Oh yes, indeed, that works. Thank you @stephg !

With this workaround the anchors not work. So o think enable fileaccess is best solution

adding the line webView.settings.setAllowFileAccess(true); under the class WebViewAndroidPlatformController along with the other settings in the file webview_android_widget.dart in the dependency plugin webview_flutter_android works, local files can be accessed
checked with version webview_flutter_android 2.8.2 & android sdk 31

You saved my life !!!

I'm using version 3.0.0 (webview_flutter_android 2.8.) and have discovered that calling controller.loadUrl() with a file scheme results in the error. However, looking at the source for loadFile(), it sets webView.settings.setAllowFileAccess(true); before calling loadUrl(). This fixes the previous version of loadFile() that did not allow accessing local files from the html page.
Thus it seems the answer to this issue is simply use loadFile().

Looks like the issue is resolved when using WebView's upstream loadFile method: https://pub.dev./packages/webview_flutter/changelog#240 This works for me - tested with some files located below getTemporaryDirectory() of path_provider package.

how to use load file method can you please suggest me

loadFile is a WebViewController method:
loadFile( String absoluteFilePath ) -> Future
Loads the file located at the specified absoluteFilePath.
The absoluteFilePath parameter should contain the absolute path to the file as it is stored on the device. For example: /Users/username/Documents/www/index.html.
Throws an ArgumentError if the absoluteFilePath does not exist.`
Code from example in plugins repo:
`Future _onLoadLocalFileExample(
WebViewController controller, BuildContext context) async {
final String pathToIndex = await _prepareLocalFile();
await controller.loadFile(pathToIndex);

}`

I am unable to find file to change this settings in current code?
Can someone please advise how to enable
webSettings.setAllowFileAccess(true);
webSettings.setAllowFileAccessFromFileURLs(true);

I am unable to find file to change this settings in current code? Can someone please advise how to enable webSettings.setAllowFileAccess(true); webSettings.setAllowFileAccessFromFileURLs(true);

You don't need to change anything
https://pub.dev/packages/webview_flutter/example
There is an example to load files from storage

I just tested the plugin example app on an android emulator running sdk 30 and it works as expected.
The webview plugin has undergone several changes since this issue was filed and since we haven't seen any comments on this issue, I will be closing this as fixed.
If anyone is still running into this issue with webview_flutter: ^4.7.0, kindly file a new issue and provide detailed steps to reproduce so we can investigate this. Thank you
Screencast from 04-12-2024 10:31:11 AM.webm

flutter doctor -v
[!] Flutter (Channel stable, 3.19.5, on Ubuntu 22.04.4 LTS 6.5.0-27-generic, locale en_US.UTF-8)
    • Flutter version 3.19.5 on channel stable at /home/daniel/fvm/versions/stable
    ! Warning: `dart` on your path resolves to /home/linuxbrew/.linuxbrew/Cellar/dart/3.2.2/libexec/bin/dart, which is not inside your current Flutter SDK checkout at /home/daniel/fvm/versions/stable. Consider adding /home/daniel/fvm/versions/stable/bin to the front of your path.
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 300451adae (2 weeks ago), 2024-03-27 21:54:07 -0500
    • Engine revision e76c956498
    • Dart version 3.3.3
    • DevTools version 2.31.1
    • If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.

[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0)
    • Android SDK at /home/daniel/Android/Sdk
    • Platform android-34, build-tools 34.0.0
    • Java binary at: /home/daniel/.local/share/JetBrains/Toolbox/apps/android-studio/jbr/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)
    • All Android licenses accepted.

[✓] Chrome - develop for the web
    • Chrome at google-chrome

[✓] Linux toolchain - develop for Linux desktop
    • Ubuntu clang version 14.0.0-1ubuntu1.1
    • cmake version 3.22.1
    • ninja version 1.10.1
    • pkg-config version 0.29.2

[✓] Android Studio (version 2023.1)
    • Android Studio at /home/daniel/.local/share/JetBrains/Toolbox/apps/android-studio
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.7+0-17.0.7b1000.6-10550314)

[✓] VS Code (version 1.87.2)
    • VS Code at /usr/share/code
    • Flutter extension version 3.86.0

[✓] Connected device (3 available)
    • sdk gphone x86 (mobile) • emulator-5554 • android-x86    • Android 11 (API 30) (emulator)
    • Linux (desktop)         • linux         • linux-x64      • Ubuntu 22.04.4 LTS 6.5.0-27-generic
    • Chrome (web)            • chrome        • web-javascript • Google Chrome 103.0.5060.53

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.


[BUG] RWX Volume Actual Size Constantly Grows with No Pod Usage

Describe the bug
Any newly created RWX PVC continually eats up space on local disk even though nothing is being added within pod volume. Pods are reporting a few kilobytes for a test text file, but inspecting the mount points on the nodes, the .img file keeps growing in size
To Reproduce
Steps to reproduce the behavior:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc-test
  namespace: test
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: longhorn-crypto-global
  resources:
    requests:
      storage: 2Ti

Expected behavior
Volume does not balloon with no disk usage from pods.
Support bundle
longhorn-support-bundle_0ade8d65-060d-4abd-9c8f-6dcdbe6a09d4_2022-05-20T14-10-34Z.zip
Support bundle
Environment

Longhorn version: 1.2.4
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Rancher Catalog App
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE2

Number of management node in the cluster: 3
Number of worker node in the cluster: 0

Node config

OS type and version: Ubuntu 22.04
CPU per node: 12
Memory per node: 8GB
Disk type(e.g. SSD/NVMe): HDD
Network bandwidth between the nodes: 1GbE

Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM
Number of Longhorn volumes in the cluster: 1

Additional context

Next one taken 2 minutes later

The accutal size is increasing initially due the fact that the filesystem (by default ext4) being initialize on the Longhorn block device. Filesystem has the metada overhead link
You should see that eventually the actual size stop increasing after this process (given that the workload pod doesn't do any io)

I retried with some smaller and larger volumes, this seems to be accurate, thanks for the update.


Universal compatibility, Focus/fade view, Bug fix

Added iPhone 6/6+ compatibility
Added iPad compatibility
Added an alert will now add a fade background as UIAlertView does
Fixed a possible crash if delegate didn't implement method
Now calling both block & delegate if set
Now calling block and/or delegate on cancel
Added delegate protocol type at initialization
Added possibility to retrieve button title from an index
Fixed layout issue when 2 buttons are shown

I close this because of new changes.


The Azure section contains information related to a deprecated portal

The section "Configure self-service password reset and other settings in Azure" contains steps that are no longer available, referencing manage.azure.com portal. This was deprecated more than 2 years ago.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 998c5ac7-53b7-548a-4f73-7f8d586abffb
Version Independent ID: 6ea1d35d-3459-34f9-aa01-e113dc7cb0c6
Content: Use the Microsoft 365 admin center to manage your subscription - Power Platform
Content Source: power-platform/admin/use-office-365-admin-center-manage-subscription.md
Service: power-platform
GitHub Login: @jimholtz
Microsoft Alias: jimholtz

@boachim
Handling via email.


Missing index.android.bundle after release build on monorepo

New Version
0.67.1
Old Version
0.64.1
Build Target(s)
Android release
Output of react-native info
System:
OS: macOS 12.1
CPU: (8) arm64 Apple M1
Memory: 103.44 MB / 16.00 GB
Shell: 5.8 - /bin/zsh
Binaries:
Node: 16.4.2 - /var/folders/y2/nc9y7hn16gj79z2lj__fl8s80000gn/T/yarn--1643282274933-0.7762066398909004/node
Yarn: 1.22.10 - /var/folders/y2/nc9y7hn16gj79z2lj__fl8s80000gn/T/yarn--1643282274933-0.7762066398909004/yarn
npm: 7.18.1 - /opt/homebrew/bin/npm
Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
Managers:
CocoaPods: 1.11.2 - /opt/homebrew/bin/pod
SDKs:
iOS SDK:
Platforms: DriverKit 21.2, iOS 15.2, macOS 12.1, tvOS 15.2, watchOS 8.3
Android SDK:
API Levels: 29, 30
Build Tools: 29.0.2, 30.0.2, 30.0.3
System Images: android-29 | Google APIs Intel x86 Atom, android-30 | Google APIs Intel x86 Atom, android-31 | Google Play Intel x86 Atom_64, android-S | Google APIs ARM 64 v8a 2, android-S | Google Play ARM 64 v8a
Android NDK: Not Found
IDEs:
Android Studio: 2020.3 AI-203.7717.56.2031.7784292
Xcode: 13.2.1/13C100 - /usr/bin/xcodebuild
Languages:
Java: 1.8.0_292 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.67.1 => 0.67.1
react-native-macos: Not Found
npmGlobalPackages:
react-native: Not Found
Issue and Reproduction Steps

Monorepo setup (yarn workspaces) is made with hoisting (all node_modules are at the top level)
React native application is under {root}/packages/mobile
/node_modules are changed to /../../node_modules at {root}/packages/mobile/android/settings.gradle, {root}/packages/mobile/android/build.gradle and {root}/packages/mobile/android/app/build.gradle
config at {root}/packages/mobile/android/app/build.gradle like below:

project.ext.react = [
    enableHermes: false,  // clean and rebuild if changing
    cliPath: "../../../../node_modules/@react-native-community/cli",
]

./gradlew bundleRelease is executed under {root}/packages/mobile/android.
index.android.bundle is missing at {root}/packages/mobile/android/app/build/generated/assets/react/release location
No error during build

No problem, right now. cliPath was wrong, it should be: "../../../../node_modules/react-native/cli",. My bad, sorry.


Have a contributor's guide

Actions (from TSC meeting 21 May)

Redraft Android Tools Setup Instructions - leads to next steps

Create a development tools setup wiki page
Create a central landing page on docs page for setting up android tools


Fix hook names

I'm not sure if we need a warning.
Probably not, no-one is reading them

Aside from conflitcs, I'm ok with it :)


-bash: -z: command not found issue

Hi
-Z command not found when I use authenticated scan. Could you kindly fix it.
docker run --rm -v $(pwd):/zap/wrk/:rw -t ictu/zap2docker-weekly zap-baseline.py
-t http://hackazon.webscantest.com/user/login
-r testreport.html
--hook=/zap/auth_hook.py \
-z "auth.loginurl=http://hackazon.webscantest.com/user/login
auth.auto
auth.username=""
auth.password="*"
auth.auto=1
auth.submit_field="loginbtn"
auth.exclude=".logout.,\Qhttp://url.com/logout\E.""
FAIL-NEW: 0	FAIL-INPROG: 0	WARN-NEW: 10	WARN-INPROG: 0	INFO: 0	IGNORE: 0	PASS: 41
OpenJDK 64-Bit Server VM warning: Exception java.lang.OutOfMemoryError occurred dispatching signal SIGHUP to handler- the VM may need to be forcibly terminated
$   -z "auth.loginurl=http://hackazon.webscantest.com/user/login \

  auth.auto
  auth.username="******" \
  auth.password="*****" \
  auth.auto=1
  auth.submit_field="loginbtn" \
  auth.exclude=".*logout.*,\Qhttp://url.com/logout\E.*""

-bash: -z: command not found

The error message you got from bash indicates that you put the -z argument on a separate line.
This works for me:
$ docker run --rm -v $(pwd):/zap/wrk/:rw -t ictu/zap2docker-weekly zap-baseline.py -t http://hackazon.webscantest.com/user/login -r testreport.html --hook=/zap/auth_hook.py -z "auth.loginurl=http://hackazon.webscantest.com/user/login
auth.auto
auth.username=''
auth.password='*'
auth.auto=1
auth.submit_field='loginbtn'
auth.exclude='.logout.,\Qhttp://url.com/logout\E.'"

Does that help?

Thanks the error was resolved. Instead of the script creating the context everytime, can we invoke the context saved in volumes.
INFO: Created user preferences directory.
Setup a new context for target http://hackazon.webscantest.com/account


SIGNATURE VERIFICATION FAILED!

SIGNATURE VERIFICATION FAILED!
Error Code: GENERIC_VERIFY_FAIL

See https://github.com/micahflee/torbrowser-launcher/issues/382#issuecomment-451603188


Switch statement mutator

As suggested by @kmdrGroch in #1078, a mutator which removes all non-default cases from a switch statement might be interesting.
My first pass is attached - feedback welcome! Working implementation for JS + TS, with tests.
We'd definitely find more mutants disabling cases one at a time, but the number of test runs needed would be enormous.
I've only tested with unit tests so far, will look into integration tests next.

@leonormes fyi

I don't quite understand it @tommilligan. In 1 test you have
it('should mutate the body of a case statement if defined as a block', () => {
      expectMutation('switch (v) { case 42: { a = "spam"; break; } }', 'switch (v) { case 42: {} }');
});

and in second
it('should remove non-default case', () => {
      expectMutation('switch (v) { case 42: a = "spam";break; }', 'switch (v) {}');
});

I also request changes for packages/stryker-mutator-specification/src/SwitchStatementMutatorSpec.ts to standardize code style.
You wrote
expectMutation('switch (v) { case 42: a = "spam";break; }', 'switch (v) {}');
then
expectMutation('switch (v) {default: a = "spam";}');
then
'switch (v) { case 0: a = "foo"; case 1: a = "qux"; break; default: a = "spam";}',
'switch (v) {default: a = "spam";}'

Hi @tommilligan thanks for the pull request! I'll have a look at it today :)

After he changes it @simondel if you like this mutator it's green to merge :)

I've made the spacing as uniform as I can while not breaking the tests - the js tests rely on string comparison which is whitespace sensitive. In my opinion the spacing is reasonable, readable and passes linting rules.
If you have a specific suggestion of which spacing to change, please do send me a patch.

@tommilligan I've made PR. Now all files of your are in same convention :)

ok, since it has to be spaced like this. @simondel ready to merge

My preference would go out to removing the switch cases one at a time.
Yes, this will create more mutants so it will take longer to run stryker, but the bennefit is also larger.
There is quite a big chance that you'll test at least one of the switch. If you remove them all at once, a test will fail but you won't know if you actually tested the cases. You only know what happens when you replace your switch statement with the code in the default statement.
I'd also be a fan of removing the default case as well in one test. That way you also know if your tests pass if you remove your default case.

sounds nice <3

Updated implementation - creates an array of mutants from a switch statement, by removing each case (including default) sequentially.

@simondel this is ready for you to review again

@tommilligan it usually takes them a while -> 1day - week. Be patient :)

@tommilligan thanks! I'll see if I can make some time this weekend :)

@simondel there is also stryker-cli PR waiting for you :)

The SwitchStatementMutator was not yet registered in the javascript mutator package. I fixed that with the latest commit.  The code looks good, but fixing survived mutants is a bit difficult since the change is made on the entire switch case:

How do you think we could fix this?

@simondel thanks for fixing the registration issue.
Updated to edit case nodes individually - the report output is a lot clearer. Tested on the stryker package itself:

Any further work needed before master?

@tommilligan thanks! I tested the TypeScript implementation yesterday and that worked like a charm 😻 I'll test the JavaScript implementation today.

Looks great! Thanks for the hard work! 🥇


new sequences appeared in the final tsv file of mmseqs cluster

Hi there,
I runned mmseqs cluster on my dataset of 5,061,557 protein sequences using these 3 command lines (MMseqs2 Version: 9f493f538d28b1412a2d124614e9d6ee27a55f45).
mmseqs createdb 3600genomes.4pub.cleaned.faa 3600genomes.4pub.cleaned.faa.mmseqsDB >mmseqs_createdb.log

mmseqs cluster 3600genomes.4pub.cleaned.faa.mmseqsDB 3600genomes.4pub.cleaned.faa.mmseqsDB_clu tmp --threads 48 -s 7.5 -c 0.5 --cov-mode 0 --max-seqs 5000 -e 0.001 --cluster-mode 0 >mmseqs_cluster.log

mmseqs createtsv 3600genomes.4pub.cleaned.faa.mmseqsDB 3600genomes.4pub.cleaned.faa.mmseqsDB 3600genomes.4pub.cleaned.faa.mmseqsDB_clu 3600genomes.4pub.cleaned.faa.mmseqsDB_clu.tsv >mmseqs_createtsv.log

When I had a look of the final tsv file (3600genomes.4pub.cleaned.faa.mmseqsDB_clu.tsv), I got two more sequences (5,061,559 instead of 5,061,557). Looking in details 4 new sequences was added:
RIFCSPHIGHO2_02_FULL_CP_45_28_RIFCSPHIGHO2_02_FULL_CP_45_28_rifcsphigho2_02_scaffold_100_1_0
RIFCSPHIGHO2_02_FULL_CP_45_28_RIFCSPHIGHO2_02_FULL_CP_45_28_rifcsphigho2_02_scaffold_100_1_1
GWA2_OP3_45_18_gwa2_scaffold_308_1_1
GWA2_OP3_45_18_gwa2_scaffold_308_1_0
whereas 2 sequences were removed:
'RIFCSPHIGHO2_02_FULL_CP_45_28_RIFCSPHIGHO2_02_FULL_CP_45_28_rifcsphigho2_02_scaffold_100_1'
'GWA2_OP3_45_18_gwa2_scaffold_308_1'
looks like mmseqs added a _0 or _1 suffix to the names of the two sequences missing. I'm pretty sure that there are no redundant sequence names in my fasta file dataset (I checked it out). Is there an explanation for these suffix ?
many thanks
Raph

MMseqs2 could until recently not handle sequences longer than 32768 residues and split longer sequences into multiple entries (with incrementing suffices _0, _1, ...).
Recently (use the latest git version), MMseqs2 learned to handle longer sequences.
By default, it will handle up to 65k residue.
By passing --dont-split-seq-by-len or --max-seq-len with a higher value to createdb it should also be able to handle those. Though, this is only a few weeks old and not very heavily battle tested yet.
Hope that helps,
Milot

sure it helps! thank you


Ensuring that PropelDateTime::getMicrotime() includes the fractional part

... to match the expected 'U.u' format in createHighPrecision(). (#1257)

Thanks!


[Snyk] Fix for 1 vulnerabilities

Snyk has created this PR to fix 1 vulnerabilities in the npm dependencies of this project.
Snyk changed the following file(s):

sdk/storage/perf-tests/storage-file-share/package.json

Vulnerabilities that will be fixed with an upgrade:

Issue
Score

Missing Release of Resource after Effective Lifetime SNYK-JS-INFLIGHT-6095116
  131

[!IMPORTANT]

Check the changes in this PR to ensure they won't cause issues with your project.
Max score is 1000. Note that the real score may have changed since the PR was raised.
This PR was automatically created by Snyk using the credentials of a real user.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.
For more information:
🧐 View latest project report
📜 Customise PR templates
🛠 Adjust project settings
📚 Read about Snyk's upgrade logic

Learn how to fix vulnerabilities with free interactive lessons:
🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

:tada: Snyk checks have passed. No issues have been found so far.
:white_check_mark: security/snyk check is complete. No issues have been found. (View Details)
:white_check_mark: license/snyk check is complete. No issues have been found. (View Details)


Object reference not set to an instance of an object. exception in AllTabPagesUpdate at 1847

Hi,
Wanna try your app but ;)
at CareLink.Form1.UpdateActiveInsulinChart(String memberName, Int32 sourceLineNumber) in C:\Users\PaulM\Source\Repos\CareLink\src\CareLink\Forms\Form1.vb:line 1961
at CareLink.Form1.AllTabPagesUpdate() in C:\Users\PaulM\Source\Repos\CareLink\src\CareLink\Forms\Form1.vb:line 1847
at CareLink.Form1.Form1_Shown(Object sender, EventArgs e) in C:\Users\PaulM\Source\Repos\CareLink\src\CareLink\Forms\Form1.vb:line 258
at System.Windows.Forms.Form.OnShown(EventArgs e)
at System.Windows.Forms.Form.CallShownEvent()
at System.Windows.Forms.Control.InvokeMarshaledCallbackDo(ThreadMethodEntry tme)
at System.Windows.Forms.Control.InvokeMarshaledCallbackHelper(Object obj)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
In Belgium on win10
I also used auto login but now it's impossible to go out from this mode, when I click ok or cancel the app close itself.
Thx for your work

@Zxquad Try <IP_ADDRESS> and if it does not fix your issue,  can you attach the Error File its under in your documents folder?

@Zxquad delete or edit CareLink.Csv  in my documents to change saved options when the app crashes but I think your crash is solved in latest update.

@paul1956 Working just fine now with <IP_ADDRESS> ;)
thx for fixing the issue ...


Add support for network rule only attachment to network policies

Is your feature request related to a problem? Please describe.
Snowflake has updated their best practice recommendation to state that allowed/blocked IP ranges should be captured in network rules that are then attached to network policies rather than specifying them directly in the network policy. This capability will eventually be deprecated by Snowflake.
Describe the solution you'd like
Update network policy object in SnowDDL to make the allowed_ip_range element optional. Add additional network_rule element to network_policy and make one or the other required.
Describe alternatives you've considered
Snowflake still supports specifying allowed_ip_range directly on the network_policy object, so the interim solution is to leave them on the network_policy.
Additional context
Recommendation from Snowflake

I'll release update for NETWORK_POLICIES soon.
Snowflake introduced a lot of additional bs around NETWORK_RULES. For example, it is not possible to drop a rule if it is connected to a network policy. But since NETWORK_RULES are schema-level objects, now it is no longer possible to drop DATABASE or SCHEMA containing such rule.
Quite annoying. I've implemented a few workaround, but not sure if it is possible to cover absolutely everything and keep reasonable performance.

@apd-bbaker , please try version 0.33.0.
Network policy config: https://docs.snowddl.com/basic/yaml-configs/network-policy
You may also use new config ACCOUNT POLICY to assign ACCOUNT-level policy: https://docs.snowddl.com/basic/yaml-configs/account-policy
Internal changes are substantial. If you notice any problems, please let me know.

@littleK0i account_policy and network_policy/rule changes are looking good. I'm running into a strange issue now with apply though where the role created specifically for snowddl to execute doesn't have the necessary permissions to create a new warehouse within the account.
I'm pretty sure I have everything correctly configured as this was working great before, but I may have been approaching it incorrectly. Content of the initial creation script to set up SnowDDL service account objects is below. Any tips on where I might be missing the mark?
/********************************************************************/
/* Set up default objects used by SnowDDL to manage the platform */
/********************************************************************/
/* Define variables for object creation */
    set role = 'platform_admin';
    set user = 'svc_manage_snowddl';
    set password = '<strong_password>';
    set warehouse = 'manage_wh';

/* Create management user and role objects */
    -- Use the built-in super user role
    use role accountadmin;

    -- Create the elevated user role
    create or replace role identifier($role);

    -- Grant elevated role to parent built-in Snowflake role
    grant role identifier($role) to role accountadmin;

    -- Grant built-in Snowflake admin roles to new role
    grant role sysadmin to role identifier($role);
    grant role securityadmin to role identifier($role);

    -- Create new admin service user
    create or replace user identifier($user)
        display_name = 'Service-Manage-SnowDDL'
        comment = "Service account used by SnowDDL to manage the Snowflake platform"
        password = $password
        default_role = $role;

    -- Grant admin role to user
    grant role identifier($role) to user identifier($user);

/* Create default user's warehouse for performing operations */
    -- Use the built-in super user role
    use role accountadmin;

    -- Create the management warehouse
    create or replace warehouse identifier($warehouse)
        warehouse_size = 'XSMALL'
        auto_suspend = 60
        auto_resume = true
        initially_suspended = true;

    -- Grant necessary access to platform role
    grant usage,operate on warehouse identifier($warehouse) to role identifier($role);

    -- Set as default wh for user
    alter user identifier($user) set default_warehouse = $warehouse;

/* Enable enhanced functionality for share management */
    -- Use the built-in super user role
    use role accountadmin;

    -- Grant necessary share management permissions
    grant create share on account to role identifier($role);
    grant override share restrictions on account to role identifier($role);

/* WARNING: Destroy platform admin objects */
    -- drop role identifier($role);
    -- drop warehouse identifier($warehouse);
    -- drop user identifier($user);

snowddl -a $SNOWFLAKE_DEFAULT_ACCOUNT -u svc_manage_snowddl -k $SNOWDDL_PRIVATE_KEY_PATH --passphrase $SNOWDDL_PRIVATE_KEY_PASSPHRASE -r platform_admin -w manage_wh --env-admin-role accountadmin --apply-unsafe --refresh-future-grants --env-prefix dev apply

2024-10-11 22:02:51.731 - INFO - Snowflake version = 8.38.3 (ENTERPRISE), SnowDDL version = 0.33.0
2024-10-11 22:02:51.731 - INFO - Account = XXXXXXXX, Region = AWS_US_WEST_2
2024-10-11 22:02:51.731 - INFO - Session =<PHONE_NUMBER>6781326, User = SVC_MANAGE_SNOWDDL
2024-10-11 22:02:51.731 - INFO - Role = DEV__PLATFORM_ADMIN, Warehouse = MANAGE_WH
2024-10-11 22:02:51.731 - INFO - Roles in session = SYSADMIN,SECURITYADMIN
2024-10-11 22:02:51.732 - INFO - ---
2024-10-11 22:02:52.245 - WARNING - Resolved WAREHOUSE [DEV__MANAGE_WH]: ERROR
(
    message   =>  SQL access control error:
Insufficient privileges to operate on account 'XXXXXXX'
    errno     =>  3001
    sqlstate  =>  42501
    sfqid     =>  01b7a08a-0003-dbf7-006f-9907010771ba
    sql       =>  CREATE WAREHOUSE "DEV__MANAGE_WH"

Which in turn appears to cause errors of this sort also:
2024-10-11 22:03:58.455 - WARNING - Resolved NETWORK_POLICY [DEV__ALLOW_LAN]: ERROR
(
    message   =>  No active warehouse selected in the current session.  Select an active warehouse with the 'use warehouse' command.

    errno     =>  606
    sqlstate  =>  57P03
    sfqid     =>  01b7a08b-0003-dbf8-006f-990701076d2a
    sql       =>  SELECT * FROM TABLE(snowflake.information_schema.policy_references(policy_name => 'DEV__ALLOW_LAN', policy_kind => 'NETWORK_POLICY'))

Script looks good.
Initial context in logs looks good as well. It shows both SYSADMIN and SECURITYADMIN roles. SYSADMIN normally includes CREATE WAREHOUSE, unless it was altered somehow. You may check grants to SYSADMIN using this command:
show grants to role sysadmin;

A few problems I can see:

Try to remove MANAGE_WH from SnowDDL config. It should be created &granted manually, and it already happens in set up script.
These options can be removed -r platform_admin -w manage_wh. Default role and default warehouse come from user parameters.
This option may potentially be removed --env-admin-role accountadmin. It is only relevant when SnowDDL admin role has ACCOUNTADMIN.

In theory, unsuccessful CREATE WAREHOUSE might be resetting current warehouse in session.
Currently we reset it back for successful CREATE WAREHOUSE commands, but not for errors. Maybe errors should be included as well.

Double-checked this.
This error does not cause current session to lose active warehouse. So it must be something else.
If issue persists after removing manage_wh from config, we may need to take a look at full log (CLI option --show-sql).

@littleK0i Found a big part of the issue. In the initial creation script the defaults assigned to the snowddl service account were not capitalized, so Snowflake apparently had an issue with that when setting them upon start of session. Corrected that, removed extraneous arguments, dropped all old objects with different ownership, and have been able to re-align everything and get a clean apply run again.

Ah, classic.
This is why SnowDDL enforces upper-cased identifiers and adds double-quotes (") for identifiers at all times. It helps to mitigate lots of non-obvious issues.


Replaced --experimental-enable-local-persistence with --persist and -persist-to

If you try to use --experimental-enable-local-persistence now, you'll get this message:
▲ [WARNING] --experimental-enable-local-persistence is deprecated.

  Use --persist in future (which will write to .wrangler/state) or
  --persist-to=./wrangler-local-state if you have existing data
  you'd like to keep using.

--persist-to takes a path relative to process.cwd() (and implies --persist, ofc).
Otherwise, --persist will use the location of the nearest wrangler.toml and derive .wrangler/state from that.

Went with .wrangler rather than hiding in node_modules as imo you really don't want to lose your local data when you hit the rm -rf node_modules button. Imo, same argument applies to the configCache stuff—we should use .wrangler/config instead of node_modules/.cache/wrangler too, just to keep all our stuff in one place. But that can be a shed for a different bike.
Thoughts on naming? The above warning messaging? Should we store some of this stuff in wrangler.toml somewhere?

What kinds of changes are you considering, and how far away are they do you think? I needed --persist-to for the D1 branch, which we will want to merge sooner or later, but doesn't need to happen yet...

Talking offline with @mrbbot, the concern is that the internal persistence format might change in a future miniflare release, and so taking it out of "experimental" before then might be premature. Which is a fair consideration, but the current format is extremely useful and both changes (move to SQLite for KV and changing filenames) should be fully automatable.
Personally, I think that leaving things as experimental doesn't help us as much as we'd want—people are still going to depend on it as it's genuinely working well right now. The thing we need is a migration strategy—a way for Miniflare vNext to detect an old-style persistence and uplift it into the new format, whatever that is.
Perhaps we make the state directory record a .version file (or equivalent, elsewhere):
.wrangler/
  state/
    .version

The contents of which could just be 1 or {"version":1}, whatever. But if that file isn't present:
if (versionFileMissing) {
  const { migrator } = await npxImport('@miniflare/persistence-migrator')
  await migrator.zeroToOne(storagePath)
}

That way we don't have to ship the migrator code with every future version of Miniflare, we can just pull it in and run it if we need it. And it gives us a way to migrate storage formats again in the future...


[6.8] [meta] bump helm support to 3.5.2 (#1065)

Backports the following commits to 6.8:

[meta] bump helm support to 3.5.2 (#1065)

jenkins test this please