Textual inversion with diffusers

I would like to get Compel working with the textual inversion and Diffusers.  I am able to get good results if I use the TextualInversionManager from InvokeAI but I don't want to add that as a dependency.
It looks like the pad tokens added in expand_textual_inversion_token_ids_if_necessary are critical.  Do you have any suggestions for how to proceed?

if you're using embeddings with a vector size of 1 then Compel doesn't need to get involved at all, they should just work OOTB.
if you have embeddings with a vector size of >1, say n=4, then you need some way of expanding the single trigger token into slots for 4 embeddings (the trigger token + 3 extras). compel/TextualInversionManager does this basically by exploding the vector size 4 TI into 4 x vector size 1 TI's.

Thanks!  The context there was very useful.  Here's an implementation of a TextualInversionManager that works with the pad tokens that diffusers create:
class DiffusersTextualInversionManager(BaseTextualInversionManager):
    def __init__(
        self,
        loader: TextualInversionLoaderMixin,
        tokenizer: CLIPTokenizer,
    ):
        self.loader = loader
        self.tokenizer = tokenizer

    def expand_textual_inversion_token_ids_if_necessary(
        self, prompt_token_ids: list[int]
    ) -> list[int]:
        if len(prompt_token_ids) == 0:
            return prompt_token_ids

        prompt = self.tokenizer.decode(prompt_token_ids)
        tokens = self.tokenizer.tokenize(prompt)
        for token in tokens:
            if token in self.tokenizer.added_tokens_encoder:
                replacement = token
                i = 1
                while f"{token}_{i}" in self.tokenizer.added_tokens_encoder:
                    replacement += f" {token}_{i}"
                    i += 1

                prompt = prompt.replace(token, replacement)

        return self.tokenizer.encode(prompt, add_special_tokens=False)

It would be nice to not go back and forth between text and tokens, but the data structures aren't setup for that yet.  Maybe this would be interesting to include in Compel?

There is a bug in diffusers that I'm working to resolve now.  After that is merged, the above can be simplified:
class TextualInversionManager(BaseTextualInversionManager):
    def __init__(
        self,
        loader: TextualInversionLoaderMixin,
        tokenizer: CLIPTokenizer,
    ):
        self.loader = loader
        self.tokenizer = tokenizer

    def expand_textual_inversion_token_ids_if_necessary(
        self, prompt_token_ids: list[int]
    ) -> list[int]:
        if len(prompt_token_ids) == 0:
            return prompt_token_ids

        prompt = self.tokenizer.decode(prompt_token_ids)
        prompt = self.loader.maybe_convert_prompt(prompt, self.tokenizer)
        return self.tokenizer.encode(prompt, add_special_tokens=False)

I'm closing the issue out as this is resolved.  Having a class in Compel or example in documentation could be useful for future devs - happy to make a PR here if you'd like.

yes please PR would be great :)

Hi @pdoane , @damian0815 ,
I'm trying to get textual inversion embeddings working in combination with compel and failing to do so.

When I keep Compel implementation, the embedding does not have any effect
When I remove Compel and use the prompt directly for inference then the embedding works

Would you have a minimal example that uses Compel and TI embeddings together please?
Thanks!
Here is what I've got:
[...]
# Load Stable Diffusion model for text-to-image
MODEL_PATH = os.getenv('MODEL_PATH')
pipeline = StableDiffusionPipeline.from_pretrained(MODEL_PATH, torch_dtype=torch.float16, revision="fp16", safety_checker=safety_checker, feature_extractor=feature_extractor)
pipeline .enable_model_cpu_offload()
pipeline .enable_xformers_memory_efficient_attention()

# Load textual inversion embeddings
EMBEDDING_PATH = os.getenv('EMBEDDING_PATH')
pipeline .load_textual_inversion(EMBEDDING_PATH, token='charturnerv2')

[...]

# Convert prompt to embedding
compel = Compel(tokenizer=pipeline.tokenizer, text_encoder=pipeline.text_encoder)
prompt_embeds = compel(prompts)

[...]
result = pipeline(
        prompt_embeds=prompt_embeds,
        negative_prompt_embeds=negative_prompt_embeds,
        width=width,
        height=height,
        num_inference_steps=steps,
        guidance_scale=guidance,
        num_images_per_prompt=1,
        generator=generators
    )

I use Compel as bellow

@alexisrolland if your prompt includes charturnerv2 and it has vector length=1 then it should just work, no need to do anything extra.

oh you may have to use a different word than charturnerv2 , see this comment: https://github.com/huggingface/diffusers/issues/3097#issuecomment-1516140736

Thanks @damian0815 but how can I know if my prompt has a vector length >1? I have used the implementation you linked to enforce charturnerv2 as trigger word, and then I'm just using compel as described above to convert a list of prompts into embeddings.

@damian0815 - charturnerv2 has vector length=15 and needs a manager like the one I provided in the PR.  It would be helpful to get that merged and then I can put a PR on the diffusers website to update the Compel examples.
@alexisrolland - The easiest way to test if you are working with a multi-vector embedding is to use pipe.maybe_convert_prompt(prompt, pipe.tokenizer)
>>> print (pipe.maybe_convert_prompt('charturnerv2', pipe.tokenizer))

charturnerv2 charturnerv2_1 charturnerv2_2 charturnerv2_3 charturnerv2_4 charturnerv2_5 charturnerv2_6 charturnerv2_7 charturnerv2_8 charturnerv2_9 charturnerv2_10 charturnerv2_11 charturnerv2_12 charturnerv2_13 charturnerv2_14

For multi-vector embeddings, diffusers creates a number of special tokens of the form f"{token}_{i}"

It would be helpful to get that merged and then I can put a PR on the diffusers website to update the Compel examples.

+1 for this, it would be fantastic!

Honestly at this point I don't know what I'm doing but for some reason it seems this worked! I have just added  prompts = pipeline.maybe_convert_prompt(prompts, pipeline.tokenizer) before converting the prompts to embeddings with Compel.
[...]
# Load model
MODEL_PATH = os.getenv('MODEL_PATH')
pipeline = StableDiffusionPipeline.from_pretrained(MODEL_PATH, torch_dtype=torch.float16, revision="fp16", safety_checker=safety_checker, feature_extractor=feature_extractor)
pipeline.enable_model_cpu_offload()
pipeline.enable_xformers_memory_efficient_attention()

# Load textual inversion embedding
EMBEDDING_PATH = os.getenv('EMBEDDING_PATH')
pipeline.load_textual_inversion(EMBEDDING_PATH, token='charturnerv2')

[...]

# Convert prompt to embedding
compel = Compel(tokenizer=pipeline.tokenizer, text_encoder=pipeline.text_encoder)
prompts = pipeline.maybe_convert_prompt(prompts, pipeline.tokenizer)
prompt_embeds = compel(prompts) # Where prompts is an array of strings containing charturnerv2

[...]
result = pipeline(
    prompt_embeds=prompt_embeds,
    negative_prompt_embeds=negative_prompt_embeds,
    width=width,
    height=height,
    num_inference_steps=steps,
    guidance_scale=guidance,
    num_images_per_prompt=1,
    generator=generators
)

maybe_convert_prompt does work in many situations but it isn't robust.  Consider having a multi-vector textual inversion named abc and a lora named abc.safetensors,  the prompt "withLora(abc.safetensors, 1)" will be incorrectly converted by maybe_convert_prompt but this works when going through the textual inversion manager interface.
You can also just copy my implementation until the PR is merged.  It's not that many lines of code :)


Can we make controlled-versions work with "*"?

I have a few dependencies that use *, for example "react": "*" or "@mycompany/mypackage": "workspace:*", and if I use the awesome controlled-version rule, it doesn't work for these because (1) they're not semver nor (2) git nor (3) file dependencies.
Would you be open to a PR adding a third function to this https://github.com/idan-at/eslint-plugin-package-json-dependencies/blob/e0dd358b5cfe9cddf77d521c618d3f31ea95967b/src/rules/controlled-versions.ts#L22-L23 something like,
const isWildcardDependency = (version: string): boolean => version.endsWith("*");

to automatically skip these?
I know I could use excludePatterns but there's enough mix-and-match in my monorepo that the dependencies have pinned versions in one workspace but wildcard versions in others 😓
Let me know if you're interested in this and I can make that PR!

Or maybe other users of this rule do NOT want * to be treated like file/git dependencies and ignored 😮? If so, perhaps we'd need another argument, something like permitWildcards?

Hey Ahmed,
controlled-versions is specifically designed to replace wildcards with specific versions. If you still want to use a wildcard, the excludePatterns is exactly for that use case :)
I’m not sure how the rule handles workspace dependencies (workspace:xx), but for that specific case we can probably add a new flag, something like ignoreWorkspacePackages. Will that help?

Thanks for the insightful feedback @idan-at!!! Your explanation makes total sense, I will add excludePatterns for the dependencies we have pure wildcards for ("react": "*").
A new flag ignoreWorkspacePackages would definitely help me but I'm honestly not sure how others structure their monorepos—I'm not sure if the best thing to offer monorepo users would be "ignore all workspace packages", because maybe the bulk of your monorepo users want to do something else. I personally can easily construct an excludePatterns string for the workspace dependencies so I have a workaround 😄

I agree, the new flag might be too broad.
I’m glad you have a workaround, closing this issue :)


Remove unused helper method in tests

Nothing calls it, and it seems relatively unlikely to be used in the future.

Pull Request Test Coverage Report for Build 504

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.02%) to 75.449%

Files with Coverage Reduction
New Missed Lines
%

internal/internal_task_pollers.go
2
73.76%

Totals

Change from base Build 498:
-0.02%

Covered Lines:
8789

Relevant Lines:
11649

💛  - Coveralls


Issue 432: Add direct import of configuration namespace to uco.ttl

This should have happened as part of resolving Issue 432.
No effects were observed on Make-managed files within UCO.  The effect that will happen in the CASE website monolithic ontology builds is an additional owl:imports triple.  The review checklist for this issue is skipped.

@eoghanscasey , @sbarnum , or @b0bkaT - this bugfix PR is ready for review.  The first to review may merge.


Deprecates Phoenix JS

What's this PR do?
Refactors lib/phoenix to use superagent to make GET requests, instead using the Phoenix JS package. This repo is the only live project that uses Phoenix JS, it's easier maintain and debug when the code lives in the repo itself.

Removes Phoenix auth credentials from .env.example- not needed now that #970 is live

Starts on tests, doesn't add unit tests for parsePhoenixCampaign or parsePhoenixError for sake of keeping this PR easier to review

How should this be reviewed?
Sanity check a few POST /chatbot, GET /campaigns, and GET /campaigns/:id requests to verify a Campaign is still fetched from the Phoenix API.
Any background context you want to provide?
When we get around to #983, we'll want to use the Phoenix GET /campaigns?ids= index query to retrieve multiple Phoenix Campaigns instead of making multiple GET /campaigns/:id requests.
Relevant tickets
#983, #970
Checklist

[ ] Tested on staging.

Codecov Report

Merging #984 into develop will increase coverage by 0.76%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop     #984      +/-   ##
===========================================
+ Coverage    58.48%   59.25%   +0.76%
===========================================
  Files           70       71       +1
  Lines         1561     1578      +17
===========================================
+ Hits           913      935      +22
+ Misses         648      643       -5

Impacted Files
Coverage Δ

lib/rogue.js
25.58% <ø> (ø)
:arrow_up:

lib/phoenix.js
96.87% <100%> (+46.87%)
:arrow_up:

config/lib/phoenix.js
100% <100%> (ø)

lib/consolebot.js
92.1% <0%> (-5.27%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a60119d...abafbe8. Read the comment docs.


Grails 3.2.9/Sec 3.1.2 got "too many redirects" and unable to access /dbconsole

I've been using the sec plugin for grails 2.x projects without problem, now trying to do a new project on Grails 3 I'm blocked on Sec config and not sure if this is a bug.
Here describes a similar problem, seems to be an issue with the hibernate plugin: http://stackoverflow.com/questions/28628398/redirect-loop-with-requestmap-with-grails-spring-security-core-2-0rc4
My case:
Created a new app, added compile 'org.grails.plugins:spring-security-core:3.1.2' at the end of my dependencies, and did:
grails s2-quickstart com.cabolabs.security User Role RequestMap

Then grails run-app.
Trying to access the /dbconsole, redirects to /login/auth
/login/auth got "localhost redirected you too many times." ERR_TOO_MANY_REDIRECTS
Added this on bootstrap, no change to the redirect problem:
for (String url in [
      '/', '/error', '/index', '/index.gsp', '/**/favicon.ico', '/shutdown',
      '/**/js/**', '/**/css/**', '/**/images/**',
      '/login', '/login.*', '/login/*',
      '/logout', '/logout.*', '/logout/*']) {
   new Requestmap(url: url, configAttribute: 'permitAll').save()
}

application.groovy generated from the quickstart script:
// Added by the Spring Security Core plugin:
grails.plugin.springsecurity.userLookup.userDomainClassName = 'com.cabolabs.security.User'
grails.plugin.springsecurity.userLookup.authorityJoinClassName = 'com.cabolabs.security.UserRole'
grails.plugin.springsecurity.authority.className = 'com.cabolabs.security.Role'
grails.plugin.springsecurity.requestMap.className = 'com.cabolabs.security.RequestMap'
grails.plugin.springsecurity.securityConfigType = 'Requestmap'
grails.plugin.springsecurity.controllerAnnotations.staticRules = [
	[pattern: '/',               access: ['permitAll']],
	[pattern: '/error',          access: ['permitAll']],
	[pattern: '/index',          access: ['permitAll']],
	[pattern: '/index.gsp',      access: ['permitAll']],
	[pattern: '/shutdown',       access: ['permitAll']],
	[pattern: '/assets/**',      access: ['permitAll']],
	[pattern: '/**/js/**',       access: ['permitAll']],
	[pattern: '/**/css/**',      access: ['permitAll']],
	[pattern: '/**/images/**',   access: ['permitAll']],
	[pattern: '/**/favicon.ico', access: ['permitAll']]
]

grails.plugin.springsecurity.filterChain.chainMap = [
	[pattern: '/assets/**',      filters: 'none'],
	[pattern: '/**/js/**',       filters: 'none'],
	[pattern: '/**/css/**',      filters: 'none'],
	[pattern: '/**/images/**',   filters: 'none'],
	[pattern: '/**/favicon.ico', filters: 'none'],
	[pattern: '/**',             filters: 'JOINED_FILTERS']
]

build.gradle
buildscript {
    repositories {
        mavenLocal()
        maven { url "https://repo.grails.org/grails/core" }
    }
    dependencies {
        classpath "org.grails:grails-gradle-plugin:$grailsVersion"
        classpath "com.bertramlabs.plugins:asset-pipeline-gradle:2.14.1"
        classpath "org.grails.plugins:hibernate5:${gormVersion-".RELEASE"}"
    }
}

version "0.1"
group "notes2"

apply plugin:"eclipse"
apply plugin:"idea"
apply plugin:"war"
apply plugin:"org.grails.grails-web"
apply plugin:"org.grails.grails-gsp"
apply plugin:"asset-pipeline"

repositories {
    mavenLocal()
    maven { url "https://repo.grails.org/grails/core" }
}

dependencies {
    compile "org.springframework.boot:spring-boot-starter-logging"
    compile "org.springframework.boot:spring-boot-autoconfigure"
    compile "org.grails:grails-core"
    compile "org.springframework.boot:spring-boot-starter-actuator"
    compile "org.springframework.boot:spring-boot-starter-tomcat"
    compile "org.grails:grails-dependencies"
    compile "org.grails:grails-web-boot"
    compile "org.grails.plugins:cache"
    compile "org.grails.plugins:scaffolding"
    compile "org.grails.plugins:hibernate5"
    compile "org.hibernate:hibernate-core:5.1.3.Final"
    compile "org.hibernate:hibernate-ehcache:5.1.3.Final"
    console "org.grails:grails-console"
    profile "org.grails.profiles:web"
    runtime "com.bertramlabs.plugins:asset-pipeline-grails:2.14.1"
    runtime "com.h2database:h2"
    testCompile "org.grails:grails-plugin-testing"
    testCompile "org.grails.plugins:geb"
    testRuntime "org.seleniumhq.selenium:selenium-htmlunit-driver:2.47.1"
    testRuntime "net.sourceforge.htmlunit:htmlunit:2.18"
    compile 'org.grails.plugins:spring-security-core:3.1.2'
}

bootRun {
    jvmArgs('-Dspring.output.ansi.enabled=always')
    addResources = true
}

assets {
    minifyJs = true
    minifyCss = true
}

Tr  to call clearCachedRequestMaps() after saving the request maps
springSecurityService.clearCachedRequestmaps()
Check http://grails-plugins.github.io/grails-spring-security-core/v3/index.html#requestmap-cache
If it does not help, please provide a sample project in github.

Yes! that broke the too many redirects. Maybe this should be added to the docs:
Bootstrap:
      for (String url in [
   	      '/', '/error', '/index', '/index.gsp', '/**/favicon.ico', '/shutdown',
   	      '/**/js/**', '/**/css/**', '/**/images/**',
   	      '/login', '/login.*', '/login/*',
   	      '/logout', '/logout.*', '/logout/*']) {
   	   new RequestMap(url: url, configAttribute: 'permitAll').save(failOnError:true)
   	}

       springSecurityService.clearCachedRequestmaps()

I also needed to add this to access the dbconsole
       new RequestMap(url: '/dbconsole/**', configAttribute: 'ROLE_ADMIN').save(failOnError:true)
Thanks!

@sdelamo please consider to update the status of this issue.
The solution you proposed should be included on the docs. Should I create another issue for that?

Thanks @ppazos
I will updated the docs in the next release. Don't need to create a new issue.


Manage a minimum version of the mobile clients

To manage a minimum version for the mobile Tchap clients (dinsic-pim/tchap-ios#214, dinsic-pim/tchap-android#493), we are working on 2 files "android-config" and "ios-config" that contain JSON data about the requirements on the client versions (we followed the example provided here: https://github.com/matrix-org/matrix-dinsic/issues/257#issuecomment-487056250)
I request by this issue to make them (when they will be ready) available from the Tchap Web client thanks to the following url:
"https://www.tchap.gouv.fr/client/config/android" -> "android-config"
"https://www.tchap.gouv.fr/client/config/ios" -> "ios-config"

@jp-modernisation-gouv-fr @Yoshiin : Do you think we should split this file in 2 (one by platform):
"android-app-version-config" and "apple-app-version-config"?

@jp-modernisation-gouv-fr @Yoshiin : Do you think we should split this file in 2 (one by platform):
"android-app-version-config" and "apple-app-version-config"?

Yes, I think we should in case one day the two versions differs.
Do I have to do the same on web client ?

@Yoshiin  FYI I updated the description of this issue
@bmarty and I are still working on the files. We will post them here as soon as they will be ready

@Yoshiin FYI I updated again the description of this issue
here are the 2 config files:
android-config.txt
ios-config.txt

@giomfo Done

@Yoshiin we will need to change the file paths. Can you please move the config files to the following path:
"https://www.tchap.gouv.fr/client/config/agent/android" -> "android-config"
"https://www.tchap.gouv.fr/client/config/agent/ios" -> "ios-config"

@Yoshiin can you please add a new config file to the following path:
"https://www.tchap.gouv.fr/client/config/secure/android" -> "secure_android-config"
secure_android-config.txt

@giomfo Done

thx

@Yoshiin Hi, we will need to update the config file with the new ones:
android-config.txt
ios-config.txt
secure_android-config.txt
And create 2 new paths:
"https://www.tchap.gouv.fr/client/config/preprod/android" -> "android-config"
"https://www.tchap.gouv.fr/client/config/preprod/ios" -> "ios-config"

Done.
FYI all config files must be in .json extension.

👍


More linters

Find more linters here:
https://github.com/alecthomas/gometalinter

Thank you, I will continue to work, but also welcome with me to improve it.

udhos probably mean that you could probably implement gometalinter support that already group and normalize output from many other linters instead of implementing them all one by one.

I understand what you mean, and I've ever thought about doing that.
But there are two reasons:

Each linters author will often change its interface and implementation
Some linters do not meet the requirements of goreporter and need to make changes

So I import a number of linters and transformed them.


modify configuration to support both elasticsearch versions 2.4 and 5.6

This WIP PR updates the elasticsearch configuration to work with both 2.4 and 5.6.
When starting the previously packed AMI ami-035f0df0234dc6d2d, elasticsearch would fail to come up and it was difficult to discover and debug errors, I discovered that they were being logged to /usr/local/var/log/elasticsearch/pelias-dev-es5-test-elasticsearch.log.
The following settings had to be changed:

bootstrap.mlockall was renamed bootstrap.memory_lock in 5.6 - I am detecting the version and using the appropriate variable name.
memlock is more difficult to achieve using systemd, some settings have been added to facilitate this.
a default config value changed and the regex had to be updated to match both #MAX_LOCKED_MEMORY=unlimited and #MAX_LOCKED_MEMORY=
index-level settings are no longer allowed in elasticsearch.yml from 5.6, so any settings beginning with index. were removed
the recommendation for these is now to use an 'index template', so I added functions to wait for ES to come up and then PUT a global index template which merges those settings with any newly generated indices, this could also be useful in the future.

Requires additional testing in 2.4 to ensure backwards compatibility is maintained and in 5.6 to ensure that this all works as expected.
Fixes https://github.com/pelias/kubernetes/issues/52
Connects https://github.com/pelias/pelias/issues/461

First of all, wow, draft pull request! Such a :sparkles: shiny :sparkles: new feature.
This looks good. I really like that this is ES5 first so we can delete some ES2 specific code when the time comes (which won't be too long).
I also like that this finally enables memlock all the way through. We were missing the systemd part so while the configuration option was set, it wasn't doing anything. Since we generally don't set up any swap with these packer/terraform scripts, it doesn't really matter, but it's good to have it working all the way:
ubuntu@ip-172-16-2-67:~$ curl "localhost:9200/_nodes?filter_path=**.mlockall"
{"nodes":{"tJbMZ6SfRYi_ExfOJIFoWg":{"process":{"mlockall":true}}}}

Onward to more testing!

okay, testing went well, caught a few bugs which have now been fixed.
I have tested the following AMIs and confirmed that they boot correctly and are correctly configured:
ami-0fbf3c6e7e6cf2db5 (es 2.4.6)
curl localhost:9200
{
  "name" : "ip-172-16-2-200",
  "cluster_name" : "pelias-dev-es5-test-elasticsearch",
  "cluster_uuid" : "XJ3zrG3WR8-wYi2fbu-cYA",
  "version" : {
    "number" : "2.4.6",
    "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd",
    "build_timestamp" : "2017-07-18T12:17:44Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.4"
  },
  "tagline" : "You Know, for Search"
}

curl localhost:9200/_template?pretty
{
  "pelias_global_settings" : {
    "order" : 0,
    "template" : "[pelias*]",
    "settings" : {
      "index" : {
        "indexing" : {
          "slowlog" : {
            "threshold" : {
              "index" : {
                "info" : "10s"
              }
            }
          }
        },
        "search" : {
          "slowlog" : {
            "threshold" : {
              "fetch" : {
                "warn" : "5s"
              },
              "query" : {
                "warn" : "5s",
                "info" : "1s"
              }
            }
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

ami-04ce9fde925957a72 (es 5.6.12)
curl localhost:9200
{
  "name" : "ip-172-16-2-73",
  "cluster_name" : "pelias-dev-es5-test-elasticsearch",
  "cluster_uuid" : "i0zhb-5RTOOKmp77jNRbAQ",
  "version" : {
    "number" : "5.6.12",
    "build_hash" : "cfe3d9f",
    "build_date" : "2018-09-10T20:12:43.732Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

curl localhost:9200/_template?pretty
{
  "pelias_global_settings" : {
    "order" : 0,
    "template" : "[pelias*]",
    "settings" : {
      "index" : {
        "indexing" : {
          "slowlog" : {
            "threshold" : {
              "index" : {
                "info" : "10s"
              }
            }
          }
        },
        "search" : {
          "slowlog" : {
            "threshold" : {
              "fetch" : {
                "warn" : "5s"
              },
              "query" : {
                "warn" : "5s",
                "info" : "1s"
              }
            }
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

both of these AMIs are currently tagged env: es2-test and env: es5-test but are now ready for planet-sized build testing.

I tested loading a full planet ES2 snapshot on the resulting ES2 AMI. After a fix to use your handy new elastic_wait script (pushed to the branch), it works fine.
Still left to do:

[ ] Test loading an ES2 snapshot onto the ES5 AMI
[ ] Test loading an ES5 snapshot onto the ES5 AMI
[ ] Benchmark a full planet ES5 build and compare performance to ES2

I don't think a cherry-pick is required, the old elastic_wait functions used %% for the curl variable (double percent) so they probably worked fine.
Double percent is mentioned in that issue as a workaround, although one commenter said it doesn't work as expected, so I'm not 100% sure.

loading snapshot portland-2019.02.19-005001-e564a08b-f72e-4b69-b824-265445b0e3b7 into ES5 worked perfectly:
curl localhost:9200
{
  "name" : "ip-172-16-1-183",
  "cluster_name" : "pelias-dev-es5-test-elasticsearch",
  "cluster_uuid" : "DxxodeR0R4uFFaArdNDuxg",
  "version" : {
    "number" : "5.6.12",
    "build_hash" : "cfe3d9f",
    "build_date" : "2018-09-10T20:12:43.732Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}

curl \
  -X POST \
  -H 'Accept: application/json' \
  -d '
{
  "aggs": {
    "sources": {
      "terms": {
        "field": "source",
        "size": 100
      }
    },
    "layers": {
      "terms": {
        "field": "layer",
        "size": 100
      }
    }
  },
  "size": 0
}' \
  'localhost:9200/pelias/_search?pretty=1&request_cache=1&preference=_replica_first'

{
  "took" : 182,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1403287,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "sources" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "openaddresses",
          "doc_count" : 1324611
        },
        {
          "key" : "openstreetmap",
          "doc_count" : 63372
        },
        {
          "key" : "transit",
          "doc_count" : 8352
        },
        {
          "key" : "whosonfirst",
          "doc_count" : 6952
        }
      ]
    },
    "layers" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "address",
          "doc_count" : 1338934
        },
        {
          "key" : "street",
          "doc_count" : 42967
        },
        {
          "key" : "stops",
          "doc_count" : 8206
        },
        {
          "key" : "venue",
          "doc_count" : 6082
        },
        {
          "key" : "locality",
          "doc_count" : 4134
        },
        {
          "key" : "neighbourhood",
          "doc_count" : 1689
        },
        {
          "key" : "postalcode",
          "doc_count" : 1037
        },
        {
          "key" : "bikeshare",
          "doc_count" : 146
        },
        {
          "key" : "county",
          "doc_count" : 75
        },
        {
          "key" : "macrohood",
          "doc_count" : 13
        },
        {
          "key" : "region",
          "doc_count" : 2
        },
        {
          "key" : "continent",
          "doc_count" : 1
        },
        {
          "key" : "country",
          "doc_count" : 1
        }
      ]
    }
  }
}

@orangejulius question about the snapshots.. I guess portland-2019.02.19-005001-e564a08b-f72e-4b69-b824-265445b0e3b7 and the other snapshots taken since ES5 became the default for docker will be ES5 indices and not backwards compatible with 2.x, or are they 2.x indices?

agh, I guess this answers my own question:
curl -sXGET 'localhost:9200/pelias/_settings?pretty&human' | jq 'to_entries[] | select(.key|startswith("pelias")).value.settings.index.version'
{
  "created_string": "5.6.12",
  "created": "5061299"
}

I'm currently loading planet-2019.01.18-003226-5a796ef5-ef44-41ad-ba01-691b8d8bde2a (a 2.x index) in to two c5d.9xlarge instances, one is running ES2.4 (i-05b14baa1ad554928) and the other is running ES5.6 (i-01a0c660e23f2fc60). Other than being different versions they are running on the same instance type and using the same code.
Interestingly the ES5 machine completed the recovery operation in ~18 minutes, while the ES2 machine seems to be only around 15% of the way through the recovery operation. So chalk that up as the first advantage of running 5.x :)

After some additional testing, this looks to work great. We've tested that all ES2 functionality still works well, that ES5 can load ES2 snapshots, perform builds, and load ES5 snapshots.
Performance testing shows a 10-20% increase in throughput across the board, as well as some latency improvements.
One notable exception was when loading an ES2-built snapshot into an ES5 cluster. In that situation we observed a considerable decrease in throughput, and a 5x increase in latency for reverse queries only. Very strange, but now we know not to use ES2-builds in an ES5 cluster.
I believe this is ready to merge!

Happy to merge 🎉


Update Jekyll configuration to not emit .html extension

A friend pointed out that our docs URLs look a bit "old school" with the html extension. There should be a simple Jekyll config that fixes this.

Here are a couple related links:
https://medium.com/@mittalyashu/how-to-remove-html-extension-in-jekyll-efeb48bd725f
https://github.com/pulumi/docs/pull/684
Summary:
Jekyll will generate links without the .html extension if they include a trailing \. A possible alternative would be creating "duplicate" S3 buckets for links with both a link.html key and a bare link key.

With #684, this should be "fixed". And it's just a matter of linking to the non-.html versions of files, and adding redirects to canonicalize the cleaner URLs. Let me see the extent to which we need to change things here, and if there is anything missing.

Looks like @cnunciato is a better owner for this in the current milestone. Here's my thinking about how to tackle this in the short-term:

Move to a world where we just emit pages as /foo/index.html instead of foo.html. See https://github.com/pulumi/docs/pull/1022 and https://github.com/pulumi/docs/pull/1014 as examples of this type of approach.

An approach we could take, but probably shouldn't:

Modify infrastructure/index.ts to make a copy of any .html files it puts into S3 and removes the extension, but sets the appropriate Content-Type header. So we create both a /foo.html and /foo object in the S3 bucket, which would allow us to serve /foo and /foo.html when deployed.
This actually works surprisingly well, but seems dubious because we would be fighting against Jekyll (which will always emit the .html file). It seems like if we wanted to do this, we should instead not serve content directly from S3, but standup some small web server to have the logic to map a request for /foo to check for an S3 object /foo, and if not present, fallback to /foo.html. (Essentially what the built-in webserver does when you run jekyll serve.)

Waiting on the outcome of #1034 on this (since it'd be handled by Hugo out of the box).

Taking this over as I've already taken care of it as part of #1088


generate mappings for 22w16a

Hi there,
I wrote a script that runs for every minecraf version and generates data from it. 22w16a got mapped with fe3a599b38f3306ee2826f528c0fcf3f416209b0, but the build failed and got fixed for 22w16b in e62e025019389979e7520f9724d78d35d2e6cb8d.
My script is now reporting erros, it is missing the mappings for that version. Could you maybe manually generate it and publish it to your servers? It is not about me being to lazy to ignore that version, I can easily exclude specific versions (https://gitlab.bixilon.de/bixilon/pixlyzer/-/blob/yarn/wrapper/PixLyzer.py#L16), but imho it should be generated for completeness.
(And yes I know, that version is broken or whatver, thats the reason for the hotfix and I should not  use it, but I am aware of that)
Thanks

Yes, I’ll get a build our for 16a shortly for completeness sake. 👍

And done! https://github.com/FabricMC/yarn/commit/e78b15e6d7b3d1d8a11bd03d51435b7516b7549c

Thanks!


Add initial filter skeleton to Listing

Co-authored-by: Howard<EMAIL_ADDRESS>Co-authored-by: Mae<EMAIL_ADDRESS>

Change display fields in list (these are not yet connected to form or db)
Add bulma classes to table
Add placeholder toggle display (control nav: list/grid/map)
Hard-coded tags and locations for now
Needs to get wired up to connect w live data and to actually filter


folder_branch_ops: in SyncAll, don't clean up file blocks

They're already handled by folderBlockOps.CleanupSyncState() (called
via the cleanup function returned by startSyncLocked()), and some
of those blocks might be re-used by subsequent retries.  So remove
them from the state passed to
folderBlockManager.cleanUpBlockState().
Issue: KBFS-2145

Could this leave blocks orphaned?

It shouldn't -- we have several tests checking that actually.  All of the blocks removed from bps will be dealt with by the folderBlockOps.CleanupSyncState() calls in the same way they've always been, even before all the SyncAll stuff was added.

Whoa this PR's old; not sure why it just showed up in my "Review requests" tab!


update docs for 1.0 release

closes #24
cc @badgley, @freeman-lab, @freyac

Let's do additional edits in follow-up PRs


[incubator][VC] add max retry rate about dws

Signed-off-by: zhuangqh<EMAIL_ADDRESS>

In our cluster, we adds initcontainers to pod and causs the pod dws fails. The failure will retry forever and let the dws slower. So we think the max retry will also add to the  dws logic.

/lgtm
/approve


Hugo cant make resources.GetRemote requests on WSL Version 2

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.101.0-466fa43c16709b4483689930a4f9ac8add5c9f66+extended linux/amd64 BuildDate=2022-06-16T07:09:16Z VendorInfo=gohugoio

Does this issue reproduce with the latest release?
Yes

Start building sites …
hugo v0.101.0-466fa43c16709b4483689930a4f9ac8add5c9f66+extended linux/amd64 BuildDate=2022-06-16T07:09:16Z VendorInfo=gohugoio
WARN 2022/07/16 00:28:16 error calling resources.GetRemote: Post "": unsupported protocol scheme ""
WARN 2022/07/16 00:28:16  error calling resources.GetRemote: Post "": unsupported protocol scheme ""

However, the issue does not occur when executing the same code Hugo.exe file generated for Windows

$hugo.exe version
hugo v0.101.0-466fa43c16709b4483689930a4f9ac8add5c9f66+extended windows/amd64 BuildDate=2022-06-16T07:09:16Z VendorInfo=gohugoio

WSL Information

PS C:\Users\benz_> wsl --status
Default Distribution: Ubuntu
Default Version: 2

Windows Subsystem for Linux was last updated on 15-07-2022
The Windows Subsystem for Linux kernel can be manually updated with 'wsl --update', but automatic updates cannot occur due to your system settings.
To receive automatic kernel updates, please enable the Windows Update setting: 'Receive updates for other Microsoft products when you update Windows'.
For more information please visit https://aka.ms/wsl2kernel.

Kernel version: 5.10.16

I am unable to reproduce the problem as described.
PS C:\WINDOWS\system32> wsl --status
Default Distribution: Ubuntu-20.04
Default Version: 2

Windows Subsystem for Linux was last updated on 7/23/2022
WSL automatic updates are on.

Kernel version: <IP_ADDRESS>

Tested with:
{{ with resources.GetRemote "https://gohugo.io/img/hugo-logo.png" }}
  <img src="{{ .RelPermalink }}" width="{{ .Width }}"  height="{{ .Height }}" alt="Hugo logo">
{{ end }}

I would tend to close this issue. Connectivity problems in WSL2 are common in all sorts of scenarios because the network setup between the WSL2 VM and the Windows host is convoluted. Wild guess: some firewall rule in the Windows host allows traffic only for one of the two calls.


Pass BuildContext to TextEditingController.buildTextSpan

Use case
I'm building a rich text editor by overriding TextEditingController.buildTextSpan in a SpannedTextEditingController. Users can apply TextAttribute to parts of the text managed by the controller (the controller maintains the active attributes as well).
The following explanation is slightly simplified
These attributes currently have a TextStyle that is applied to the TextSpans created in TextEditingController.buildTextSpan:
class TextAttribute {
  TextStyle get style;
}

However, I would like for the attributes to be themeable. E.g. a 'Header' attribute should not have a fixed size. They'd have a resolve method that takes a BuildContext - so they can use InheritedWidgets - and returns the actual TextStyle:
class TextAttribute {
  TextStyle resolve(BuildContext context);
}

Of course now I need access to the wrapping BuildContext in buildTextSpan.
Alternatives
The SpannedTextEditingController could hold theming data itself (for example with an AttributeTheme field). I do not think this is a great solution because:

The responsibility of retrieving the theme is placed outside the SpannedTextEditingController.
It does not match with the way themes are used in flutter. Themes are not stored for later use, instead they are depended on and used immediately.
The theme would need to be depended upon (AttributeTheme.of) where the controller is created, which is not necessarily close to the widget that displays the text. This would most likely be a source of confusion and error as theming beneath the context that the controller was created in is not applied.

Proposal
Add a BuildContext parameter to TextEditingController.buildTextSpan and pass the build context in EditableTextState.build.
This would be a breaking change for inheritors and users of the API (because the parameter would be required).

My input:
I think your  current text style property can be merged with a new text style from Theme:
    final myTextStyle = TextStyle();
    final newTextStyle = myTextStyle.merge(
        Theme.of(context).textTheme.bodyText1
    );

And then provide the newTextStyle to the TextEditingController.buildTextSpan method. Does that works?

Thank you for the quick response! :)
I wouldn't pass TextStyle to the controller, because the controller itself is agnostic towards which attributes it can apply.
Instead I'd have a custom theme class that has a Map that attributes can use to retrieve theming data (based on some naming convention, e.g. BoldAttribute would use attributeTheme['bold_weight'] to store its FontWeight).
Passing this theming data to the constructor of my custom controller is how I would do it in the alternative discussed in my initial post. It has a few downsides that I would like to not have.
I depend on buildTextSpan being called by EditableText so I cannot customize what parameters are passed in my code.

Reopening since https://github.com/flutter/flutter/pull/72344 was reverted.  Will be fixed again in https://github.com/flutter/flutter/pull/73510.

Reopening since https://github.com/flutter/flutter/pull/72344 was reverted.  Will be fixed again in https://github.com/flutter/flutter/pull/73510.


Input field Optional[OneToManyInput] is a required field in the resulting schema

Not sure if this is expected behaviour, but if I define an input type like this:
@gql.django.input(ServiceInstance)
class ServiceInstancePartialInput(NodeInput):
    id: Optional[gql.ID]
    service: Optional[OneToManyInput]

The resulting schema marks 'service' as required field:

~I can pass an empty object into service, since set is not required, but I'd expect service to be not required instead.~
Edit: the schema allows for it, but if I do that the mutation fails.

Hey @gersmann ,
Hrm, I'm trying to replicate the issue here but it is working fine for me. The schema shows as not required and I can not send service at all to the input and it will work. It will not work if the field cannot be null though, for obvious reasons.
Can you provide more information regarding this? Like, the traceback itself, how are you doing the mutation, etc?

@bellini666 you're right, not sure how that schema came to be, I didn't reproduce the problem correctly. It happens when inheriting a base type, this should reproduce it:
@gql.django.input(ServiceInstance)
class ServiceInstancePartialInput(NodeInput):
    id: gql.ID
    service: Optional[OneToManyInput]
    key: Optional[str]

@gql.django.input(Connector)
class ConnectorPartialInput(ServiceInstancePartialInput):
    pass

@gql.type
class ServiceMutation:
    update_connector: ConnectorType = gql.django.update_mutation(ConnectorPartialInput)

{
  "data": null,
  "errors": [
    {
      "message": "Field 'ConnectorPartialInput.service' of required type 'OneToManyInput!' was not provided.",
      "locations": [
        {
          "line": 18,
          "column": 25
        }
      ],
      "path": null
    }
  ]
}

Stack:
GraphQL request:18:25
17 | mutation updateConnector {
18 |   updateConnector(input:{id: "961", key:"foo_bar"}) {
   |                         ^
19 |     __typename
Stack (most recent call last):
  File<EMAIL_ADDRESS>line 890, in _bootstrap
    self._bootstrap_inner()
  File<EMAIL_ADDRESS>line 932, in _bootstrap_inner
    self.run()
  File<EMAIL_ADDRESS>line 870, in run
    self._target(*self._args, **self._kwargs)
  File<EMAIL_ADDRESS>line 683, in process_request_thread
    self.finish_request(request, client_address)
  File<EMAIL_ADDRESS>line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File<EMAIL_ADDRESS>line 747, in __init__
    self.handle()
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/servers/basehttp.py", line 178, in handle
    self.handle_one_request()
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/servers/basehttp.py", line 201, in handle_one_request
    handler.run(self.server.get_app())
  File<EMAIL_ADDRESS>line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/contrib/staticfiles/handlers.py", line 76, in __call__
    return self.application(environ, start_response)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/wsgi.py", line 133, in __call__
    response = self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/base.py", line 130, in get_response
    response = self._middleware_chain(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/deprecation.py", line 117, in __call__
    response = response or self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/middleware.py", line 57, in __call__
    response = toolbar.process_request(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/timer.py", line 65, in process_request
    return super().process_request(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/headers.py", line 50, in process_request
    return super().process_request(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/staticfiles.py", line 117, in process_request
    return super().process_request(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  [Previous line repeated 1 more time]
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/logging.py", line 77, in process_request
    return super().process_request(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/debug_toolbar/panels/__init__.py", line 193, in process_request
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/Workspaces/parcellab/pl_manage/common/middlewares.py", line 16, in __call__
    return self.get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/Users/gersmann/Workspaces/parcellab/pl_manage/pl_manage/views.py", line 57, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/strawberry/django/views.py", line 162, in dispatch
    result = self.schema.execute_sync(
  File "/Users/gersmann/.local/share/virtualenvs/pl_manage-Ql6hhOMP/lib/python3.8/site-packages/strawberry/schema/schema.py", line 182, in execute_sync
    self.process_errors(result.errors, execution_context=execution_context)
2022-02-21 16:03:37,926 INFO django.server "POST /graphql-next HTTP/1.1" 200 190

Noticed that inheritance behaves a bit strange in general, for instance if case a Django foreign key relationship is inherited, it is represented by the generic DjangoModelType, but if it is defined directly on the child type, it is represented by the correct concrete GraphQL type.

Hey @gersmann ,
You are calling your types Partial, but you are using gql.django.input on them instead of gql.django.partial (or even gql.django.input(partial=True)). That may be the issue with it being required.

Noticed that inheritance behaves a bit strange in general, for instance if case a Django foreign key relationship is inherited, it is represented by the generic DjangoModelType, but if it is defined directly on the child type, it is represented by the correct concrete GraphQL type.

Don't know if I understood what you meant here... Could you provide an example of that too?

@bellini666
sorry for the screenshots, haven't been able to export the schema with django yet (apps aren't ready...)
So this definitiion:
@gql.django.type(Customer)
class CustomerType:
    id: gql.auto
    name: gql.auto

@gql.django.type(ServiceInstance)
class ServiceInstanceType:
    customer: CustomerType

@gql.django.type(Connector)
class ConnectorType(ServiceInstanceType):
    pass

@gql.type
class Query:
    service_instances: List[ServiceInstanceType] = gql.field()
    connectors: List[ConnectorType] = gql.field()

Gives me for ServiceInstanceType:

But for ConnectorType:

Thats what I meant this inheritance behaving a bit strange.

This here replicates the original issue, also seems like it is inheritance-related:
@gql.django.input(ServiceInstance)
class ServiceInstanceInput:
    service: Optional[OneToManyInput]

@gql.django.input(Connector)
class ConnectorInput(ServiceInstanceInput):
    updated_at: gql.auto

@strawberry.type
class Mutation:
    create_service_instance: ServiceInstanceType = gql.django.create_mutation(ServiceInstanceInput)
    create_connector: ConnectorType = gql.django.create_mutation(ConnectorInput)

ServiceInstanceInput:

ConnectorInput:

Hey @gersmann ,
Think I found the issue. Should be fixed in the release I'm doing right now. Please reopen this issue if it is not

@bellini666 Thanks a lot.


pretrain dataset

Hi, I download about 38k pdb files by using the config files, and paper indicates the pretrain dataset is 805k. Should this be expected? Many thanks!

Hi! Since the AF DB is too large to directly load in the CPU memory, we instead do a Round-Robin pretraining on each species. https://github.com/DeepGraphLearning/GearNet/blob/09a4b92602aae261d10c18d9613b99ccc5a6ebc6/script/pretrain.py#L60-L68
If you don't specify a species, it will download each species one by one and pretrain our model on it. The species and split id can be passed when loading AF DB (https://torchdrug.ai/docs/api/datasets.html#alphafolddb).


Fix flaky health trait test

What happened?
While running this test,
I noticed the Integration's Ready condition is true for a while before becoming false,
which in turns makes this assertion fail sometimes,
making the test flaky.
- firstTruthyTime: "2023-11-20T13:02:00Z"
  lastTransitionTime: "2023-11-20T13:02:00Z"
  lastUpdateTime: "2023-11-20T13:02:00Z"
  message: 1/1 ready replicas
  reason: DeploymentReady
  status: "True"
  type: Ready

Steps to reproduce
Run this e2e test a few times until it fails.
Relevant log output
apiVersion: camel.apache.org/v1
kind: Integration
metadata:
  annotations:
    camel.apache.org/operator.id: test-305d00bf-726e-4415-9567-2573502fa0bf
  creationTimestamp: "2023-11-20T12:49:24Z"
  generation: 1
  name: never-readypnryt
  namespace: test-305d00bf-726e-4415-9567-2573502fa0bf
  resourceVersion: "19430"
  uid: aa15f5f4-1343-4d20-b61c-48cc8f187298
spec:
  sources:
  - content: |
      /*
       * Licensed to the Apache Software Foundation (ASF) under one or more
       * contributor license agreements.  See the NOTICE file distributed with
       * this work for additional information regarding copyright ownership.
       * The ASF licenses this file to You under the Apache License, Version 2.0
       * (the "License"); you may not use this file except in compliance with
       * the License.  You may obtain a copy of the License at
       *
       *      http://www.apache.org/licenses/LICENSE-2.0
       *
       * Unless required by applicable law or agreed to in writing, software
       * distributed under the License is distributed on an "AS IS" BASIS,
       * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
       * See the License for the specific language governing permissions and
       * limitations under the License.
       */

      import org.apache.camel.builder.RouteBuilder;

      public class NeverReady extends RouteBuilder {
          @Override
          public void configure() throws Exception {
              from("timer:tick").id("never-ready")
                  .to("controlbus:route?routeId=never-ready&action=stop&async=true")
                  .setHeader("m").constant("string!")
                  .setBody().simple("Magic${header.m}")
                  .log("${body}");
          }
      }
    name: NeverReady.java
  traits:
    health:
      enabled: true
      livenessProbeEnabled: false
      livenessScheme: HTTP
      readinessProbeEnabled: true
      readinessScheme: HTTP
      startupProbeEnabled: false
      startupScheme: HTTP
status:
  conditions:
  - firstTruthyTime: "2023-11-20T12:49:24Z"
    lastTransitionTime: "2023-11-20T12:49:24Z"
    lastUpdateTime: "2023-11-20T12:49:24Z"
    message: test-305d00bf-726e-4415-9567-2573502fa0bf/test-305d00bf-726e-4415-9567-2573502fa0bf
    reason: IntegrationPlatformAvailable
    status: "True"
    type: IntegrationPlatformAvailable
  - firstTruthyTime: "2023-11-20T12:49:24Z"
    lastTransitionTime: "2023-11-20T12:49:24Z"
    lastUpdateTime: "2023-11-20T12:49:24Z"
    message: explicitly disabled by the user
    reason: masterTraitConfiguration
    status: "True"
    type: TraitInfo
  - firstTruthyTime: "2023-11-20T13:01:56Z"
    lastTransitionTime: "2023-11-20T13:01:56Z"
    lastUpdateTime: "2023-11-20T13:01:56Z"
    message: kit-cldlbl2hqaes73e5b8j0
    reason: IntegrationKitAvailable
    status: "True"
    type: IntegrationKitAvailable
  - firstTruthyTime: "2023-11-20T13:01:56Z"
    lastTransitionTime: "2023-11-20T13:01:56Z"
    lastUpdateTime: "2023-11-20T13:01:56Z"
    message: deployment name is never-readypnryt
    reason: DeploymentAvailable
    status: "True"
    type: DeploymentAvailable
  - firstTruthyTime: "2023-11-20T13:02:00Z"
    lastTransitionTime: "2023-11-20T13:02:00Z"
    lastUpdateTime: "2023-11-20T13:02:00Z"
    message: 1/1 ready replicas
    reason: DeploymentReady
    status: "True"
    type: Ready
  dependencies:
  - camel:controlbus
  - camel:timer
  - mvn:org.apache.camel.k:camel-k-runtime
  - mvn:org.apache.camel.quarkus:camel-quarkus-java-joor-dsl
  - mvn:org.apache.camel.quarkus:camel-quarkus-microprofile-health
  digest: vZi_hPRbPLDXMUuj2SPoFJLW2OPdK7PcY7M79Rh_eGB8
  image: <IP_ADDRESS>/test-305d00bf-726e-4415-9567-2573502fa0bf/camel-k-kit-cldlbl2hqaes73e5b8j0@sha256:15df1691ea5f08295324eb828228c69335bfe56ef4c544645e7813ad829539e5
  integrationKit:
    name: kit-cldlbl2hqaes73e5b8j0
    namespace: test-305d00bf-726e-4415-9567-2573502fa0bf
  lastInitTimestamp: "2023-11-20T12:49:24Z"
  observedGeneration: 1
  phase: Running
  platform: test-305d00bf-726e-4415-9567-2573502fa0bf
  profile: Kubernetes
  replicas: 1
  runtimeProvider: quarkus
  runtimeVersion: 3.2.0
  selector: camel.apache.org/integration=never-readypnryt
  version: 2.2.0-SNAPSHOT

Camel K version
2.1.0

Would someone please update the labels on this issue?

@rinaldodev thanks for reporting. What labels do you think it is required? Please also let me know if you are going to work on this so I can assign to you.

@squakez I believe that one is fine. Yes, you can assign it to me, thanks.

To bring more clarity to this issue: this appears to be caused by the camel-routes health check reporting as Ready before the routes had some time to load. See https://issues.apache.org/jira/browse/CAMEL-20242.

@rinaldodev are you planning to work on this for next 2.4 release? Should I add it to the milestone?

Some more analysis on this flaky test can be found in #5345 which is a duplicate of this one

Fixed with #5346


ion-datetime: the displayed time and the picker time are off by 1 hour

Ionic version:  (check one with "x")
[ ] 1.x
[x] 2.x
I'm submitting a ...  (check one with "x")
[x] bug report
[ ] feature request
[ ] support request
Current behavior:
Set the displayed time in ion-datetime to 12:01 AM and tap/click on the control. The picker time is showing 01:01 AM
Expected behavior:
If the displayed time in ion-datetime is 12:01 AM, the picker should also show 12:01 AM
Steps to reproduce:
This issue is reproducible using the demo from the Docs

Go to the official ion-datetime page
Select 12:01 AM for the Tokyo datetime picker under World Times (as shown below)

Tap/click on the ion-datetime control again and you'll see the picker time is off by 1 hour

Ionic info: (run ionic info from a terminal/cmd prompt and paste output below):
Your system information:

Cordova CLI: 6.4.0
Ionic Framework Version: 2.0.0-rc.3-201612021933
Ionic CLI Version: 2.1.13
Ionic App Lib Version: 2.1.7
Ionic App Scripts Version: 0.0.46
ios-deploy version: Not installed
ios-sim version: Not installed
OS: Windows 10
Node Version: v6.7.0
Xcode version: Not installed

Any updates on this issue? Any workarounds?

Should be fixed in 2.3 release!


Loop seems to do the wrong thing when reservoir reports 0

It is known that there is give or take 10 units after the reservoir reports 0 units left.  But loop looks like it assumes no delivery while insulin is being delivered and increases the dose.  I think the safest bet is to either listen for the no delivery alert or to assume delivery and disable looping until the reservoir is changed.
I just noticed while dropping 0.7units/5min(around 7mmol/L at the time), no carb on board, at was dosing 3.4u/hr when I normally take 1.2u/hr at that time.
Almost looks like a feedback loop.

This is fixed here: https://github.com/loudnate/Loop/pull/146
cc: @sarahspins


varLib doesn't correctly interpolate corner masters

Noto Sans Oriya has the following master configuration:

Regular (red)
Thin (blue)
Thin Condensed (purple)
Bold (green)
Bold Condensed (orange)

This provides enough information to generate a Condensed instance at wght=0 wdth=-1, but as shown above, at wght=0 only the regular master has any support, so the Condensed instance comes out as equal to the Regular. (Unless you use mutatormath, which works fine.)
My feeling is this: varLib.model has a good enough model for the mathematically constrained task of generating deltas for variation stores; the mistake is using it for the more general task of interpolating UFOs across a designspace, which is a different interpolation task. So, do we want to:

just flat out not support this kind of master configuration (sorry, designers!)?
find a way to generalise varLib.model so that it can cope with it?
use a different interpolation model for fontmake.instantiator?

I can't really tell from the image where the masters are.

Masters are at
Thin: Weight=26 Width=100
Black: Weight=170 Width=100
Condensed Extra Thin: Weight=26 Width=68
Condensed Extra Black: Weight=170 Width=68
Regular: Weight=78 Width=100
Desired instance is at:
Condensed: Weight=26 Width=100

the desired instance is at the same location as the Thin master (Weight=26 Width=100) ?
what is the intended default location in this system?

Sorry, desired should be Weight=78 Width=68. Regular is the default.
And I was wrong about my hunch. This isn't just a UFO interpolation issue; a variable font generated from these masters will also have a problems due to there being zero support for the other masters. So it is a varLib problem.

In what model do these master work? What makes it a varlib problem and not a design problem?

Duovars without monovars ain’t never gonna work.

But it does work with mutatormath.

It also works correctly when exporting from Glyphs. Maybe better to say that varLib is the only model where it doesn’t work!

Well great export it from Glyphs, or have them publish ther code.

It might be nice in this case if the missing monovar could be generated from two duovars — i.e. Condensed being a (100-26)/(170-26) interpolation between Condensed Thin and Condensed Black. I’m not sure how generalizable that approach is though, or whether it’s what Glyphs does. Although an improvement on an absent monovar, it won’t yield a result that any designer would have drawn at that location, unless by chance, as the interpolation ratio claims a relationship between either side of default weight that does not exist.

I'm sorry, what's a "monovar"/"duovar"?

Sorry, it’s @dberlow’s terminology.  It helps avoid ambiguity of “corner master” in fonts of >2 axes, and it’s a concise way of talking about the support that masters need: trivars rely on a complete set of duovars and monovars.

Picking at the "it works in mutatorMath":

If I build a designspace with mutatorMath it will select Weight:26, Width:100 as its default, not Weight 78, Width 100. And then the Weight 78, Width 100 master, in gray, will limit the influence of  the red and blue masters. There will not be a 50% mix of the blue and red at Weight 78, Width 68. Probably the instance Weight: 78, Width: 68 will look like Weight 78, Width 100. This is not a good solution, but it questions "it works"

In comparison, if I remove the Weight 78, Width 100 master, the red and blue will mix as expected. This is of course no solution either, but I guess it is what you expect for Weight 78, Width 68 and you can use that as a support.

https://user-images.githubusercontent.com/1243961/194878085-a0f00ad6-85b5-4a49-af40-7d88c902068e.mp4

From what I can see this issue is not that different from the issue from 2017. And the solution would be similar as well: you need to sample the value at Weight 78, Width 68 from a space without Weight 78, Width 100 and then insert that as a master.

it is a good idea to plan a designspace before starting work

Probably the instance Weight: 78, Width: 68 will look like Weight 78, Width 100. This is not a good solution, but it questions "it works"

I just tried it again; it definitely looks like it's doing something sensible to derive an ExtraCondensed from the master layout it's given. In FontTools, ExtraCondensed looks like Regular, but in Mutatormath it looks like a condensed version...

it is a good idea to plan a designspace before starting work

Well, true. But some of us are engineers and we have to handle what we are given.
And when designers hand us designspaces that work fine in other software, it's not unreasonable for them to expect it to work in the free software pipeline as well. They don't know it's impossible, so they try it and find that it works. It's only when we get hold of it that it turns out to be impossible.

As Erik suggested, varLib and MutatorMath use the same algorithm, it's just that MutatorMath is picking a different base master here.
varLib is designed to produce OpenType-compatible deltas. We can have a higher-level model in varLib that synthesizes missing masters.

Isn't it just the choice of default that causes the problem? It would work just fine if one of the corners were tge default. Seen from that angle, the problem is equivalent to L4 instantiation.

Isn't it just the choice of default that causes the problem?

Yeah but in a font we want the Regular to stay the default.

Yeah but in a font we want the Regular to stay the default

Sure, all I'm saying is that we could start with the wrong default, and fix it with L4. Perhaps there's a better way, it just occurred to me that L4 is one way to solve it.

I don't think there's anything actionable here. Feel free to disagree.


@mui/codemod@next v5.0.0/preset-safe smDown -> xlDown

Using codemod preset-safe gives wrong translation for smDown.

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯
After running npx @mui/codemod@next v5.0.0/preset-safe src/
The following change happens:

Expected Behavior 🤔
I expected smDown to be translated to mdDown

@afreakk Did you run npx @mui/codemod@next v5.0.0/preset-safe src/ once or multiple times? I checked on my machine and it seems to work correctly (<Hidden smDown => <Hidden mdDown)
FYI, the preset-safe should be used once as noted in the migration docs. If you found error or need to run preset-safe again, ideally you need to clear the local changes otherwise you will encounter the issue like this.
from the hidden-down-props codemod inside preset-safe, it will transform
  xsDown => 'smDown',
  smDown => 'mdDown',
  mdDown => 'lgDown',
  lgDown => 'xlDown',
  xlDown => 'xlDown',

so if you if you run multiple times, it can end up from smDown to lgDown.

Yes, I ran it twice because i got errors on the first run.
Thats the reason


Missing NuGet and PSGallary Required for Registration

Please add NuGet and PSGallary to this list as it is required to be able to install the registration module as stated this Doc https://learn.microsoft.com/en-us/azure-stack/hci/deploy/register-with-azure#register-a-cluster-using-powershell

When you register an Azure Stack HCI cluster you have to run Install-Module -Name Az.StackHCI which goes to NuGet to get the required modules for the registration. If we are not able to download the modules, we will not be able to register the HCI cluster with Azure.
The link I provided shows a note that NuGet and PSGallary are required.
Note
You may see a prompt such as Do you want PowerShellGet to install and import the NuGet provider now? Answer Yes(Y).
If you see another prompt saying Are you sure you want to install the modules from 'PSGallery'? Answer Yes(Y).

Add NuGet and PSGallery to the Required firewall URLs table

You can find the NuGet URLs and ports documented in the Visual Studio and Azure Services behind a firewall or proxy server
https://learn.microsoft.com/en-us/visualstudio/install/install-and-use-visual-studio-behind-a-firewall-or-proxy-server?view=vs-2022

No I am not asking for the Visual Studio article to be updated. I am asking for the missing NuGet URL's to be added to the Firewall requirements for Azure Stack HCI article (https://learn.microsoft.com/en-us/azure-stack/hci/concepts/firewall-requirements. The Visual Studio article has the NuGet URLs listed that need to be also added to the Firewall requirements for Azure Stack HCI.

Got some clarification -- #please-open

@DellProSupportGse
Thanks for bringing this to our attention.
I'm going to assign this to the document author so they can take a look at it accordingly.

#assign: @cosmosdarwin

#assign: @SaibabaBalapur-MSFT
The issue is on https://learn.microsoft.com/en-us/azure-stack/hci/concepts/firewall-requirements, I'm not the author of this doc. Please reassign. Thanks!

@alexbuckgit
Could you please look into this.

This issue tags the incorrect article. We are addressing the issue and it's being worked on in a PR in the private repo. #please-close


Support reading scalar variables through the zarr plugin

Reading scalar variables through the zarr plugin is erroring with a ValueError: Invalid chunk_id for numpy array: 0. Should have been: (). This is due to ((0,) * da.ndim) being () when ndims = 0. In turn, that does not match the requested chunk (0,) so the exception is raised.. This removes the invalid chunk_id check when dealing with scalar variables (ndim = 0). The added test fails without the change.

Sure, I added a scalar variable the generic dataset that everything used so the other tests were exposed to a scalar variable. I actually expected more to fail :smiley:!


Use flash_attention for SD2

What does this PR do?
Use flash attention for SD2 base.
Before submitting

[ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
[ ] Did you read the contributor guideline?
[ ] Did you read our philosophy doc (important for complex PRs)?
[ ] Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
[ ] Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
[ ] Did you write any new necessary tests?

Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@zpcore sd 2 base doesn't require attention masks. What is the reasoning for adding it?

You are right, the attention mask is always None. I don't know why the original code pass  None mask to the attention. To be safe I just keep it there.

@zpcore sd 2 base doesn't require attention masks. What is the reasoning for adding it?

You are right, the attention mask is always None. I don't know why the original code pass None mask to the attention. To be safe I just keep it there.

I believe it was added because there is a method to use masking for diffusion models, but this particular model and implementation we're targeting doesn't use it. LGTM.


0.61.0 - Jest "Cannot find module ..."

After upgrading from 0.60.5 to 0.61.0, when I run Jest tests I will get errors anywhere I was mocking a component or native module from React Native using the syntax where I mock the module/component by name rather than providing a path to it or mocking all of React Native, i.e:
jest.mock('TextInput', () => {
  ...
})

This has always worked in previous versions of React Native and is still mentioned as a valid way to mock things in the jest docs.
It looks like I can fix the issue by mocking all of React Native and overriding the specific modules I care about, however I'm curious if this change was intentional or not.  If it is, then I can move this issue to the Jest repo and let them know they need to update their docs.
React Native version:
System:
OS: macOS 10.14.6
CPU: (8) x64 Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
Memory: 1.01 GB / 16.00 GB
Shell: 3.2.57 - /bin/bash
Binaries:
Node: 8.11.3 - ~/.nvm/versions/node/v8.11.3/bin/node
Yarn: 1.16.0 - /usr/local/bin/yarn
npm: 5.6.0 - ~/.nvm/versions/node/v8.11.3/bin/npm
Watchman: 4.9.0 - /usr/local/bin/watchman
SDKs:
iOS SDK:
Platforms: iOS 13.0, DriverKit 19.0, macOS 10.15, tvOS 13.0, watchOS 6.0
Android SDK:
Build Tools: 25.0.2
IDEs:
Android Studio: 3.5 AI-191.8<IP_ADDRESS>91312
Xcode: 11.0/11A420a - /usr/bin/xcodebuild
npmPackages:
react: 16.9.0 => 16.9.0
react-native: 0.61.1 => 0.61.1
npmGlobalPackages:
react-native-cli: 2.0.1
Steps To Reproduce

Using a 0.61.0 build of RN, write a Jest test(doesn't have to assert anything in particular).
In that test, mock a React Native library component by adding jest.mock('View')
Run the test

Describe what you expected to happen:
Jest should fail with an error telling you Cannot find module 'View' ...

Similar to: https://github.com/facebook/react-native/issues/26447

This is intentional and you need to mock modules the same way as any other JS module now. You could in theory specify the path to the TextInput module, but the path is a private implementation detail that could change between releases. The logically correct approach is to code to the interface and mock out react-native.

@fossage Could you please show what you solution ended up being?
My use case I want to check that Keyboard's dismiss was called. So far I can just do:
jest.mock('Keyboard', () => ({
  dismiss: jest.fn(),
}));

jest.mock('react-native', () => ({
  Keyboard: { dismiss: jest.fn() }
}));

Works to mock the Keyboard, but obviously the other react-native libraries, eg. Image are undefined now. Would I need to add them to the mock?

@ccfz , unfortunately I was just assuming that I could get this working without having actually tried.  I never was able to figure out a solution, specifically due to the problem you are describing.
The things I did try to no avail were:

Updating the moduleMapper field in jest.config.js to map names like Keyboard to the location of the underlying libraries.  This failed for numerous reasons.
Doing something like:

jest.mock('react-native', () => {
  const actualRN = require.requireActual('react-native')
  return {
    ...actualRN,
    Keyboard: {
      ...
    }
  }
})

This failed because we would end up getting the actual version of RN rather than the mocked version created by the react-native Jest preset.(makes sense but I was curious to try).  If you change the require.requireActual('react-native') to just require('react-native'), you end up in an endless loop(also makes sense but figured I would try anyway).
Ultimately I decided to fallback to RN 0.60.6 until some solution is found:(.

I was able to successfully mock the individual modules by mocking the direct path in react-native so items like
jest.mock('Keyboard', () => {})
jest.mock('TextInput', () => {})

can become
jest.mock('react-native/Libraries/Components/Keyboard/Keyboard', () => {})
jest.mock('react-native/Libraries/Components/TextInput/TextInput'), () => {})

@lroling8350  Mocking via the direct path is definitely a way to make this work. The problem with that is when RN changes the file path like @ide pointed out above. Checking if Keyboard's dismiss was called can also be tested using a spy if someone just wants to do that.
However, I have more complicated mocks related to Animated and TextInput that require actual mocking. @fossage I ran into the same problem trying to mock out react-native. @ide could you maybe go into a little more detail about how something like jest.mock('Animated') should be mocked with 0.61.1? It's a common mock that requires an overriding aspect of Animated. Here is a sample mock I took from here, but we have very similar code in our company's app.
jest.mock('Animated', () => {
  const ActualAnimated = require.requireActual('Animated');
  return {
    ...ActualAnimated,
    timing: (value, config) => {
      return {
        start: (callback) => {
          value.setValue(config.toValue);
          callback && callback()
        },
      };
    },
  };
});

This seems to be the issue with the most details related to mocking in RN 0.61.1, @fossage maybe re-open the issue for now?

Yes, @fossage please re-open this issue.
We too have not yet been able to upgrade to 0.61 because of the mocking issues.
This is how we mock analytics-react-native as suggested here:
import {NativeModules} from 'react-native';

NativeModules.RNAnalytics = {};

const mockAnalytics = jest.genMockFromModule('@segment/analytics-react-native');
jest.mock('@segment/analytics-react-native', () => mockAnalytics);

Which raises the following error:
● Test suite failed to run

    Failed to load Analytics native module.

      at Object.<anonymous> (node_modules/@segment/analytics-react-native/build/cjs/bridge.js:6:11)
      at Object.<anonymous> (node_modules/@segment/analytics-react-native/build/cjs/analytics.js:38:16)
      at Object.<anonymous> (node_modules/@segment/analytics-react-native/build/cjs/index.js:3:19)
      at Object.<anonymous> (jest/setup.js:22:26)

Same issue here nothing is working using latest rn 0.61.1

In case this is helpful for anyone, we successfully mocked out LayoutAnimation using the code below
jest.mock("react-native/Libraries/LayoutAnimation/LayoutAnimation", () => ({
  ...require.requireActual(
    "react-native/Libraries/LayoutAnimation/LayoutAnimation"
  ),
  configureNext: jest.fn(),
}));

@SophieDonut Thank you for the example! A caveat with your solution is that the path to the library is a private implementation and therefore not stable. If RN changes the path to the libraries then you would have to update all your mocks. See @ide comment at the top.

Mocking something like the libraries and components is fine when you point to the component directly as above, but when trying to mock something like BackHandler or TouchableOpacity it seems to throw an issue. These mocks come from the @types folder and cant be mocked seemingly.

A caveat with your solution is that the path to the library is a private implementation and therefore not stable..

I'm aware it's definitely not the best solution and everyone should be aware that if they use it it could break at any time but for now it seems to be the only way to get the tests running on the new React Native version. We tried to mock all of React Native the way we're currently mocking out Layout Animation but it didn't seem to work properly...
If anyone manages to successfully create a mock it would be great if they could share it here.

We use jest.mock('NativeModules', () => { [...] to mock only the native system code of a module.  Mocking the entire module is not an option because we need to test the JS layer glue wrapping the native code.
Not being able to do such is preventing our upgrade to 0.61.x.

@ccfz You'd mock out react-native and re-export it with a different value for Animated. For example, mockReactNative = { ...originalReactNative, Animated: { whatever } }. Similar to how you're mocking out just Animated.timing, you would mock out just ReactNative.Animated. Same thing for NativeModules, etc.

@ccfz that does not work for me. Not sure if this is what you meant, but here is what I tried:
jest.mock('react-native', () => {
  const actualRn = jest.requireActual('react-native')
  return {
    ...actualRn,
    TouchableOpacity: () => 'TouchableOpacity',
  }
})

results in an error:
TypeError: (0 , _codegenNativeComponent.default) is not a function

I also tried it with require.requireActual(...) with the same result.
So far the only viable solution seems to be importing the react-native modules via the full path (Which is -like mentioned before- not stable)
Thank you for looking into this

Have you been able to successfully mock the TouchableOpacity or the BackHandler? As I have not been able to just yet.

I got same error as @0akl3y people think it's just easy to mock react-native but it's not! So please stop giving us those boilerplate answers.

@ide I think the issue here is that doing what you are proposing does not work.  Doing so will lead to the error that @Oakl3y pointed to earlier.  Many of us on the thread have tried it, and numerous other variations, and haven't come up with any viable solution.  I still have a few things I would like to try, however I have a feeling that this is something that will need to be addressed by the RN team.

@fossage This is a working example that mocks a NativeModule and parts of a JS module (LayoutAnimation). It lets you import a pre-mocked instance of react-native, override whatever you'd like, and then access those mocks from your tests:
jest.config.js
module.exports = {
  preset: 'react-native',
  setupFiles: ['<rootDir>/setup.js'],
};

setup.js
import * as ReactNative from 'react-native';

jest.doMock('react-native', () => {
  ReactNative.LayoutAnimation.configureNext = jest.fn();
  ReactNative.NativeModules.Override = {great: 'success'};
  return ReactNative;
});

tests/example-test.js
import { LayoutAnimation, NativeModules } from 'react-native';

test(`NativeModules contains overrides`, () => {
  expect(NativeModules.Override).toEqual({ great: 'success' });
});

test(`other mock NativeModules are preserved`, () => {
  expect(NativeModules.Timing).toBeDefined();
  expect(NativeModules.Timing.createTimer).toBeDefined();
  expect(NativeModules.Timing.createTimer.mock).toBeDefined();
});

test(`LayoutAnimation.configureNext is mocked`, () => {
  expect(LayoutAnimation).toBeDefined();
  expect(LayoutAnimation.configureNext).toBeDefined();
  expect(LayoutAnimation.configureNext.mock).toBeDefined();
});

Output
$ jest
 PASS  __tests__/App-test.js
  ✓ NativeModules contains overrides (18ms)
  ✓ other mock NativeModules are preserved (4ms)
  ✓ LayoutAnimation.configureNext is mocked (3ms)

Test Suites: 1 passed, 1 total
Tests:       3 passed, 3 total
Snapshots:   0 total
Time:        7.328s
Ran all test suites.

The main limitation that comes to mind is that you need to use the same doMock call for all tests and Jest's flexibility often offers ways to work around that.

@ide That does not work for component getters, since it seems like those cannot be overwritten that way. However there is a way around this by using Object.defineProperty:
import React from 'react'
import renderer from 'react-test-renderer'

jest.mock('react-native', () => {
  const ReactNative = jest.requireActual('react-native')
  return Object.defineProperty(ReactNative, 'Button', {
    get: jest.fn(() => {
      return 'MockedButton'
    }),
  })
})

it('Mocks the component via react-test-renderer and correctly displays the mock in the snapshot', () => {
  const { Button } = require('react-native')
  const button = renderer.create(<Button title="it-also-works-in-snapshots" />)
  expect(button.mock).toBeDefined
  expect(button).toMatchSnapshot()
})

Test results:
 PASS  src/components/__tests__/example.test.js
  ✓ Mocks the component via react-test-renderer and correctly displays the mock in the snapshot (6ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   1 passed, 1 total
Time:        0.559s, estimated 1s
Ran all test suites matching /example.test/i.

Watch Usage: Press w to show more.

The snapshot also renders the mock correctly:
// Jest Snapshot v1, https://goo.gl/fbAQLP

exports[`Mocks the component via react-test-renderer and correctly displays the mock in the snapshot 1`] = `
<MockedButton
  title="it-also-works-in-snapshots"
/>
`;

The following article pointed me in the correct direction:
https://bambielli.com/til/2017-09-17-jest-doesnt-mock-getters/
However this seems like a workaround and requires much refactoring. I would prefer if it just worked like before. Publishing  private implementation details seems to be the lesser evil than to lose the capability to conveniently mock those dependencies.

However there is a way around this by using Object.defineProperty:

This looks reasonable -- due to the way RN has worked for a long time, Object.defineProperty is one of the few ways to override a getter (Object.setPrototypeOf({ Button: 'MockedButton' }, ReactNative) being another). I updated the code sample above so the examples are in one place.

Publishing private implementation details seems to be the lesser evil than to lose the capability to conveniently mock those dependencies.

For a given version of RN, code like this will work:
jest.doMock('react-native/Libraries/Components/Button', () => 'MockedButton');

It is a more brittle approach because of the coupling between your test and the internal path. If you understand it introduces coupling and can fix the tests if needed in the future, this approach might work for you. But in general with Jest, mocking the public interface you expect to interact with is a more robust approach and more likely to last.

I've found a workaround for this issue. You can add all Library subdirectories of react-native to the moduleDirectories prop in your jest.config.js. I've done this for our app with a recursive file-walker function:
// jest.config.js
const fs = require('fs')
const path = require('path')

function walk(dir) {
  let files = fs.readdirSync(dir)
  files = files.map(file => {
    const filePath = path.join(dir, file)
    const stats = fs.statSync(filePath)
    if (stats.isDirectory()) return walk(filePath)
    else if (stats.isFile()) return path.dirname(filePath)
  })

  let fileMap = files.reduce((all, folderContents) => all.concat(folderContents), [])
  fileMap = fileMap
    .filter((filePath, index) => fileMap.indexOf(filePath) === index)
  return fileMap
}

const moduleDirectories = [
  'node_modules',
  ...walk(__dirname + '/node_modules/react-native/Libraries'),
]

module.exports = {
   ...
   moduleDirectories,
}

After that, all mocks resolve correctly.

@fossage I think all the open questions have been answered and the issue can be closed.
Thanks again @ide for your detailed example that really helped.

@ide , thanks for the help!

The easiest way I found to mock out the part that I needed was to just import it and overwrite it.

Previously I had this
jest.mock('Animated', () => {
  const ActualAnimated = require.requireActual('Animated')
  return {
    ...ActualAnimated,
    timing: (value, config) => {
      return {
        start: (callback) => {
          value.setValue(config.toValue)
          callback?.({ finished: true })
        },
      }
    },
  }
})

To fix the issue in 0.61
import { Animated } from 'react-native'

Animated.timing = (value, config) => {
  return {
    start: (callback) => {
      value.setValue(config.toValue)
      callback?.({ finished: true })
    },
  }
}

Hi all!
After applying any and all of workarounds proposed here I still face the issue with these errors:
Test suite failed to run

    Invariant Violation: TurboModuleRegistry.getEnforcing(...): 'DeviceInfo' could not be found. Verify that a module by this name is registered in the native binary.

      at invariant (node_modules/invariant/invariant.js:40:15)
      at Object.getEnforcing (node_modules/react-native/Libraries/TurboModule/TurboModuleRegistry.js:39:3)
      at Object.<anonymous> (node_modules/react-native/Libraries/Utilities/NativeDeviceInfo.js:45:48)
      at Object.<anonymous> (node_modules/react-native/Libraries/Utilities/Dimensions.js:15:1)

Any clue how to resolve it?

Hi all!
After applying any and all of workarounds proposed here I still face the issue with these errors:
Test suite failed to run

    Invariant Violation: TurboModuleRegistry.getEnforcing(...): 'DeviceInfo' could not be found. Verify that a module by this name is registered in the native binary.

      at invariant (node_modules/invariant/invariant.js:40:15)
      at Object.getEnforcing (node_modules/react-native/Libraries/TurboModule/TurboModuleRegistry.js:39:3)
      at Object.<anonymous> (node_modules/react-native/Libraries/Utilities/NativeDeviceInfo.js:45:48)
      at Object.<anonymous> (node_modules/react-native/Libraries/Utilities/Dimensions.js:15:1)

Any clue how to resolve it?

Hi @G0retZ just the error message makes it hard to pinpoint the issue. Could you please show the mocking you did and or test. That way we can help you figure out what went wrong.

@fossage This is a working example that mocks a NativeModule and parts of a JS module (LayoutAnimation). It lets you import a pre-mocked instance of react-native, override whatever you'd like, and then access those mocks from your tests:
jest.config.js
module.exports = {
  preset: 'react-native',
  setupFiles: ['<rootDir>/setup.js'],
};

setup.js
import * as ReactNative from 'react-native';

jest.doMock('react-native', () => {
  // Extend ReactNative
  return Object.setPrototypeOf(
    {
      // Redefine an export, like a component
      Button: 'MockedButton',

      // Mock out properties of an already mocked export
      LayoutAnimation: {
        ...ReactNative.LayoutAnimation,
        configureNext: jest.fn(),
      },

      // Mock a native module
      NativeModules: {
        ...ReactNative.NativeModules,
        Override: {great: 'success'},
      },
    },
    ReactNative,
  );
});

__tests__/example-test.js
import { LayoutAnimation, NativeModules } from 'react-native';

test(`NativeModules contains overrides`, () => {
  expect(NativeModules.Override).toEqual({ great: 'success' });
});

test(`other mock NativeModules are preserved`, () => {
  expect(NativeModules.Timing).toBeDefined();
  expect(NativeModules.Timing.createTimer).toBeDefined();
  expect(NativeModules.Timing.createTimer.mock).toBeDefined();
});

test(`LayoutAnimation.configureNext is mocked`, () => {
  expect(LayoutAnimation).toBeDefined();
  expect(LayoutAnimation.configureNext).toBeDefined();
  expect(LayoutAnimation.configureNext.mock).toBeDefined();
});

Output
$ jest
 PASS  __tests__/App-test.js
  ✓ NativeModules contains overrides (18ms)
  ✓ other mock NativeModules are preserved (4ms)
  ✓ LayoutAnimation.configureNext is mocked (3ms)

Test Suites: 1 passed, 1 total
Tests:       3 passed, 3 total
Snapshots:   0 total
Time:        7.328s
Ran all test suites.

The main limitation that comes to mind is that you need to use the same doMock call for all tests and Jest's flexibility often offers ways to work around that.

That was a good solution ;) cheers.

Mocking React Native's interface did not work for me, as @altany suggested. I created a setup.js and added it to setupFilesAfterEnv config option:
jest.mock('react-native/Libraries/Utilities/Platform', () => ({
    OS: 'ios',
    select: jest.fn((selector) => selector.ios),
}));

Then, in each test, you can change Platform's implementation like so:
import { Platform } from 'react-native';

test('your test', () => {
    Platform.select.mockImplementation((platforms) => platforms.android);
	Platform.OS = 'android';
	...
});

Thanks a lot. It is Sunday and I am doing this.

I tried all the solutions above in React Native version 0.63.4, but that does not work I want to changes the Platform's implementation in each test.

@mksglu I'm yet to try the approach I suggested before in 0.63.4. Did you manage to find a solution meanwhile?

Hello @acostalima, thank you for the response. I guess finally found a solution for 0.63.4. I created a new file in __mocks__ directory named is react-native.js and the file includes of above lines:
import * as ReactNative from 'react-native';

jest.doMock('react-native', () => {
  return Object.setPrototypeOf(
    {
      Dimensions: {
        get: () => ({width: 414, height: 896}),
      },
      Platform: {
        ...ReactNative.Platform,
        OS: 'ios',
        select: jest.fn((selector) => selector),
      },
      NativeModules: {
        ...ReactNative.NativeModules,
        SettingsManager: {
          settings: {
            AppleLocale: 'en_US',
          },
        },
        I18nManager: {
          localeIdentifier: 'en_US',
        },
      },
    },
    ReactNative,
  );
});

Then, when I create a test file like button.test.js:
import {Platform} from 'react-native';
describe('Button', () => {
  it('should call onPress function', () => {
    Platform.OS = 'android';
  });
});

@mksglu thanks! 🙇 🙏  I'll take note of this approach and try to make it work on my end on 0.63.4.


Storage Account Count for subscription

azure.StorageAccounts.Manager.Usages.List().First().CurrentValue appears to be counting only classic storage accounts. Is there a better way to get just count of all storage account in a subscription without listing them (zure.StorageAccounts.List().Count()) ?

@lucasmaj can you provide the full name of the package you are using, also provide the version number you are using.
Any information to reproduce this error will be helpful.

using Microsoft.Azure.Management.Fluent;
using Microsoft.Azure.Management.ResourceManager.Fluent;
Microsoft.Azure.Management.Fluent 1.16.0
Microsoft.Azure.Management.Storage.Fluent 1.16.0

Thank you for explaining. I see now. That is because the limit for storage accounts is per Subscription-Location?

It appears that to get ListByLocation method we need to use Microsoft.Azure.Management.Storage package 9.0.0-preview?
Also what is the simplest way of getting ServiceClientCredentials from service principal creds? Currently getting exception:
Microsoft.Rest.Azure.CloudException: Response from service 'Microsoft.Storage' does not contain sufficient information to enforce access control policy.

@lucasmaj
We plan to close the issue as not hear from you for a long itme.
Feel free to let us know if you need any futher assistant.
@shahabhijeet
Would you please help to close the issue, or give the permission to close the issue.

Closing per @blueww 's comments


Allow monday as first weekday

The picker is great but in some cases (different locales) I need to start the week from Monday.
As of now, I have to edit the source code (replace .startof('week') with 'isoweek'). It would be nice if there was an option for that.

Hi @Ishy-Fishy, first day of the week depends on the locale you set.
Actually there's no way to change this behavior, but if there's real need I can implement an ad-hoc feature to let you choose which day should be shown as first in the Month View.
Does your scenario really need to overcome the default/manual locale setting?

Isn't anything critical, but it would be nice to have. For some countries that aren't in the locale list

Well, ok I can add this feature. Just to know, which country are you interested in? :)

A few countries (like Bosnia & Herzegovina) that don't provide enough userbase to warrant a full webpage translation but should have the EU locale preset, en_IE doesn't work for some reason :(

From Moment docs, seems pertinent, hope it helps!:
As of 2.13.0 you can pass a bool as the first parameter of the weekday functions. If true, the weekdays will be returned in locale specific order. For instance, in the Arabic locale, Saturday is the first day of the week, thus:
moment.locale('ar');
moment.weekdays(true); // lists weekdays Saturday-Friday in Arabic
moment.weekdays(true, 2); //will result in Monday in Arabic
Note: Absent the locale specific parameter, weekdays always have Sunday as index 0, regardless of the local first day of the week.

@Ishy-Fishy Did you ever get round to fixing this? If not, did you add the moment locale script <script src="//cdnjs.cloudflare.com/ajax/libs/moment.js/2.10.6/moment-with-locales.js"></script>.
@indrimuska I think it may be worth adding a note to to the locales section reminding people to add this. I didn't spot that this file was required and couldn't figure out why the locales weren't changing the picker UI.