How to build for

Hi, if your library depends on an external package, how can you build for <script type="module">? Bonus points if you can make it work as CJS, too.
Here's our library
// my-library.js
import fetch from 'isomorphic-unfetch'
export default function (url) {
  // do stuff
  return fetch(url)
}

Using rollup es format, can this work?
<script type="module">
  import myLibrary from './dist/my-library.es.js'
  myLibrary(42)
</script>

Right now you it warns you
Uncaught TypeError: Failed to resolve module specifier "isomorphic-unfetch". Relative references must start with either "/", "./", or "../".`

If you change the library's import to something like ./node_modules/isomorphic-unfetch/es-version.js it'll work locally but of course break the built version. What to do? What is this rabbit hole?
Any help would be great appreciated! Thank you.
Possibly related

https://github.com/rollup/rollup/issues/1803
https://stackoverflow.com/questions/49947250/how-do-rollup-externals-and-globals-work-with-esm-targets

Hey 🙂 I'm kinda new to rollup but also ran into this problem. The way I got around it is to use @rollup/plugin-node-resolve as a plugin when bundling modules for use in the browser, so that dependancies are all bundled up and no resolving is needed client-side. Something like:
// rollup.config.js
{
      input: 'src/main.js',
      output: {
          file: pkg.module,
          format: 'esm'
      },
      plugins: [
          resolve(),
      ]
},

Whereas if I was building out an es module to be used in Webpack / whatever I wouldn't bundle the dependancies and let the build tool handle it.
There might be a better way of handling this so if anyone has a better option give me a shout :)
Cheers!

@oskarrough it's not possible right now to change where the identifier points to.
In the future, we could probably use import-maps.

For me, it was a matter of changing:
from:

to:


Fail loudly when 'text' key is missing.

No one especially me would like to know how much time this took to figure out :)

Sorry, this PR was meant as a kind of sketch of the intention. But I take that there are other uses of message dicts that do not need the 'text' key. My understanding of ASGI is far too little for me to fix this problem, but I guess if there's a way to make it fail loudly when supposedly mandatory elements of the message dict are missing, that'd be good.
It could also be that different message type object classes and constructors would be the way forward?

This belongs in Daphne and has been fixed by @andrewgodwin -- thanks!


How to properly use ButtonBehavior

Hi, I can't figure out how to properly bind to on_press or on_release events when extending Label (or probably any other Widget) with ButtonBehavior:
import kivy
from kivy.app import App
from kivy.lang import Builder

kivy.require('1.9.0')

kv = """
GridLayout:
    cols: 1
    spacing: 5
    MyLabel:
        text: "Label #1"
    MyLabel:
        text: "Label #2"
    MyLabel:
        text: "Label #3"
    MyLabel:
        text: "Label #4"
    MyLabel:
        text: "Label #5"

<MyLabel@Label+ButtonBehavior>:
    on_press: print(self.text)
    on_release: print(self.text)
    canvas.before:
        Color:
            rgb: 0.2, 0.6, 0.6
        Rectangle:
            pos: self.pos
            size: self.size
"""

class ButtonBehaviorTestApp(App):
    def build(self):
        return Builder.load_string(kv)

if __name__ == '__main__':
    ButtonBehaviorTestApp().run()

Complete output from the console is:
[INFO   ] [Logger      ] Record log in /Users/martin/.kivy/logs/kivy_15-06-01_89.txt
[INFO   ] [Kivy        ] v1.9.0
[INFO   ] [Python      ] v2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]
[INFO   ] [Factory     ] 173 symbols loaded
[DEBUG  ] [Cache       ] register <kv.lang> with limit=None, timeout=None
[DEBUG  ] [Cache       ] register <kv.image> with limit=None, timeout=60
[DEBUG  ] [Cache       ] register <kv.atlas> with limit=None, timeout=None
[INFO   ] [Image       ] Providers: img_tex, img_imageio, img_dds, img_gif, img_sdl2 (img_pil, img_ffpyplayer ignored)
[DEBUG  ] [Cache       ] register <kv.texture> with limit=1000, timeout=60
[DEBUG  ] [Cache       ] register <kv.shader> with limit=1000, timeout=3600
[DEBUG  ] [App         ] Loading kv <./buttonbehaviortest.kv>
[DEBUG  ] [App         ] kv <./buttonbehaviortest.kv> not found
[INFO   ] [OSC         ] using <multiprocessing> for socket
[INFO   ] [Window      ] Provider: sdl2
[INFO   ] [GL          ] OpenGL version <2.1 INTEL-10.6.20>
[INFO   ] [GL          ] OpenGL vendor <Intel Inc.>
[INFO   ] [GL          ] OpenGL renderer <Intel HD Graphics 4000 OpenGL Engine>
[INFO   ] [GL          ] OpenGL parsed version: 2, 1
[INFO   ] [GL          ] Shading version <1.20>
[INFO   ] [GL          ] Texture max size <16384>
[INFO   ] [GL          ] Texture max units <16>
[DEBUG  ] [Shader      ] Fragment compiled successfully
[DEBUG  ] [Shader      ] Vertex compiled successfully
[DEBUG  ] [ImageImageIO] Load </Applications/Kivy.app/Contents/Resources/kivy/kivy/data/glsl/default.png>
[INFO   ] [Window      ] auto add sdl2 input provider
[INFO   ] [Window      ] virtual keyboard not allowed, single mode, not docked
[INFO   ] [Text        ] Provider: sdl2
[DEBUG  ] [Resource    ] add </Library/Fonts> in path list
[DEBUG  ] [Resource    ] add </System/Library/Fonts> in path list
[DEBUG  ] [Resource    ] add </Users/martin/Library/Fonts> in path list
[DEBUG  ] [Resource    ] add </Applications/Kivy.app/Contents/Resources/kivy/kivy/data/fonts> in path list
[DEBUG  ] [Base        ] Create provider from mouse
[INFO   ] [Base        ] Start application main loop
[INFO   ] [GL          ] NPOT texture support is available

None of the events is triggered.

Try making the ButtonBehavior the first superclass, not the second,

Nice, this was really the issue. I think this should be mentioned also in the documentation because usualy you first define superclass and then its mixins. Doing it the other way around wouldn't occure to me.

It is already mentioned in the documentation:

The behavior class must always be before the widget class. If you don’t specify the inheritance in this order, the behavior will not work because the behavior methods are overwritten by the class method listed first.

Ou, you're right. How could I miss it.


RDF endpoint?

might be connected to https://github.com/elexis-eu/lexonomy/issues/61
I'm wondering if it would be possible to automate an RDF export of the headwords, definitions, etc. using the Ontolex vocabulary? I think it could have two aspects:

a global RDF export of a dictionary
an RDF export of each entry accessible at the entry URL when requested with either a special URL (adding .ttl at the end for instance) or using the headers (the Cool URI principles)

What do you think?

This is currently not doable because Lexonomy doesn't actually "know" where the headwords, definitions etc. are. All Lexonomy "sees" inside the entries is XML elements.
(By the way, this feature request is not related to #61. #61 is about an API in Sketch Engine, which is something else entirely.)

Ontolex RDF export is supported to a certain extent - Lexonomy is inducing headwords and senses based on well-known schemas and TEI-Lex0


[WIP] Add macOS 10.14 Dark Mode support

WIP

As suggested in #276 by @jansol and @nangtrongvuon, I'm creating a PR so we can push the discussion further. Bear in mind this is my first contribution to a Swift project. 🙈
This is a work in progress, and should probably not be merged as-is.
Set theme flow
The Theme changes originally follow this flow during the app lifecycle:

Select theme

App initialization

Get theme list from core
Get current theme from preferences

Manual theme setting

Get theme name from menu item

Apply theme

Send a set theme message to core
core responds to mac with a JSON-like theme object
The provided theme object is parsed by mac and sets the menu item as active

Working stuff
What works in the current PR, is selecting the macOS theme after app initialization. Doing so will inhibit the set theme message usually sent to core, and update the UI with a Theme object directly.
Not working stuff
I found that setting the theme this way does not work for some keys (they are explicitly listed in comments in the code).
Opening the app when macOS theme is selected results in an error from core, because of the set theme message sent on startup with a theme name that does not exist in core. We could inhibit this as well.
core seems to still have a word to say in what's displayed on the screen somehow, and the previously selected theme will apply to the editor text color.
Thoughts
As you can see, this PR is pretty hacky, so we may want to come up with a better solution. But I feel this is definitely doable.
I feel the best solution (for a working v1) would be to have a dedicated theme menu entry that is treated like a special case that sends a set theme message to core with an existing theme (e.g. base16-ocean.light) based on the current selected system color scheme, so the editor can use the text color from core, and then set the rest of the app UI with a Theme object using semantic NSColors.
If any of you guys have insights or general tips regarding Swift development please feel free to share your feedback and contribute!
😃

Example of use when the previously active theme clashes:

I think this should use the vibrant blending mode, covered in the WWDC video, starting from around 28:30 -- it's supposed to avoid this kind of issues.
Besides that, I think making this a special theme is the wrong approach. Instead I would make it a flag that when enabled uses the theme colors and vibrant blending to tint the transparent window. It should be fine even when switching between dark and light, but in case it's not (and even if it is, because this would be quite cool) there could be an option to specify two active themes (dark and light) and have xi-mac automatically switch when the system appearance changes.

Okay, I haven't dug into the mojave docs/talks yet so I defer to @jansol's suggestion!

@cmyr I'm glad I came out with the same analysis as your high-level thought!
I originally had in mind what you just said in your for now paragraph. I was going for the option to enable the system dark/light mode, which would use one of core's themes for each corresponding state and override the actual app properties.
I also like the long term idea you have, to bundle 2 special themes for each corresponding system theme. However there's a few things I don't get. First, I couldn't find out how/where themes are stored/defined in core. I obviously need to understand this in order to bundle themes from xi-mac. Also, I guess a theme (syntax highlighting-speaking) has a fair amount of colors that have to work in harmony. I wouldn't say I'm a bad UI designer, but there's a high chance people are used to be able to choose their syntax-highlighting independently of their app UI.
What I'm thinking is similar to what @jansol said, we could consider the best of both worlds:

Allow users to set a flag that they want the UI to match the system light/dark setting
Allow them to choose which syntax highlighting they want to use independently

Here's a mockup of how the user could use this:

Light system theme
Dark system theme

Thinking long-term, we could imagine that users could import their own themes, flag them as made for dark or light backgrounds, and xi-mac can automagically use them depending on the system's preferences.
However, I do not think the vibrant blending mode is quite suitable for the use case we have. As I understand it, vibrant blending mode should be used to modify colors that are displayed on top of a potentially changing background color. So one might think it would be useful to update the syntax highlighting colors, but there's a few issues:

vibrant blending mode should not be used for non-grayscale colors (read: do not use vibrant blending mode on colored items)
the only colors we should override are the background, gutter, and default text. Using semantic NSColors is IMHO the way to go there.

Do you guys think the two active themes and the flag solution is the way to go there?

Do you guys think the two active themes and the flag solution is the way to go there?

I believe so. An interesting thing to try would be to allow the user to specify just one theme and then automatically invert its colors (or maybe just their brightness/value?) for the other appearance. Some syntax themes also work quite well on both bright and dark backgrounds, especially when the default color is automatically updated to match (Monokai for example looks quite nice against either background) so automatic inversion should be optional too, if it is supported. This is of course an optional bit of functionality at this stage.

While inverting colors would keep the same contrast compared to the other type of background, it also breaks any color semantics the theme could've set. The rule of thumb here would be to invert only the grayscale colors, because they're supposedly the ones that are going to lack contrast against an inverted background. But again, there's no guarantee that it will actually look nice. I'm not a fan of trying to guess what the theme should look like. While I agree some themes can look good on either backgrounds, I still think that being able to choose one light and one dark theme would be better for the user.

A drive-by comment: something else to keep in mind is the "unified_titlebar" feature, and how that works with this?

also @habovh there's some info on how themes work here: https://github.com/xi-editor/xi-editor/issues/883

I guess unified titlebar should update according the system theme if the flag is set to follow the system theme, otherwise we can still update it according to the theme if it isDark?
@cmyr thanks for the heads up! Will definitely read this!

While inverting colors would keep the same contrast compared to the other type of background, it also breaks any color semantics the theme could've set.

Hence value invert aka keep the hue & saturation but invert the lightness.
Anyway this is mostly a "could be nice" gimmick for people who are too lazy to define two separate themes.

Oh sorry @jansol I didn't get the invert lightness only 🙈 My bad, this is definitely a could be nice feature!

@habovh @jansol is there consensus here? I'm happy to more or less stay out of it, if you two agree on an approach you have my blessing. 🙈

I think so. To recap:

Have a flag for switching between theme-defined background color and NSAppearance
User can select separate light and dark syntax themes
Leaving one of those unset causes xi to derive it from the other one by inverting the lightness of the entire palette

What I'm not sure is what the best notation for this would be in the config.. Simply presence/absence of theme_light/theme_dark keys? Using a separate theme key? That one would require the lightness to be tagged directly in the theme, which would require some more refactoring.

We could just let the user pick whatever theme they want for 'light' and 'dark' and if it looks bad that's just user error?

Would it be too much work for each theme to define a light and dark variant?

It's mostly a question of knowing whether lightness should be inverted for the light or the dark variant or neither. It could be configured like this:
theme_light = "Tomorrow Night Eighties"
theme_light_invert = "true"
#theme_dark = "Tomorrow Night Eighties"
theme_dark_invert = "false"

We'll want some slightly more sophisticated logic for defaults though:

If a theme variant is specified, default to invert = "false"
If a variant is not specified, default to the other variant

if invert is not specified, default to "true"

This way it is enough to specify theme_light = "InspiredGitHub" and you get a reasonable default for both light and dark variants.
@ollieatkinson For bundled themes that is a reasonable thing to do. However, if you found a nice theme online and wanted to use that you'd end up having to manually create a copy and replace every single color. So yes, I'd say that would be needlessly much work, at least if automatic inverting turns out to look anywhere near as good as I expect.

@habovh What's the status on this? Are you blocked on anything?

@habovh no worries! this is definitely not critical right now, if you get around to it eventually that's great, if someone else grabs it that's great too.

@habovh I'm going to close this for now; if you're interested in picking it up again at some point feel free to reopen!


Story & Theme

Purgatory. Civilization is a land without spirit, and the people here are basically walking dead. The player character has started to wake up, and realize there is more. The monsters and threats beyond the wall are manifestation of his own doubt and fear. Tethering is giving in to this fear. Banishment can be delayed by encountering beauty or hope.
CS Lewis story for inspiration.
Deific influence in regions?

End point: wizard gives up power to join Lenore
What do you fall in love with? Superficial stuff, convenience, something deeper
Maybe he "kills" beast versions and has a realization
Rilke quote about dragons and princesses

Starts as pure revenge quest, extermination. A clue is found that Lenore can be found. Goal changes


⚠ Bofrost.at has degraded performance

In 0681835, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1828 ms

Resolved: Bofrost.at performance has improved in df7b26c.


how to auto migrate tables with circular relationships

How to auto migrate tables with circular relationships ?

The document you expected this should be explained
type Product struct {
	gormext.Model
	StoreID              uuid.UUID  `gorm:"type:uuid REFERENCES stores(id);not null" validate:"required"`
	MainVariantID   *uuid.UUID `gorm:"type:uuid REFERENCES variants(id)"`
}

type Variant struct {
	gormext.Model
	ProductID uuid.UUID `gorm:"type:uuid REFERENCES products(id);not null" validate:"required"`
}

Expected answer
AutoMigrate(new(Product), new(Variant))

maybe follow https://stackoverflow.com/questions/13413158/how-do-i-create-a-circular-foreign-key-dependency-in-postgresql


Adding flag NOGIT to io-package.json required

You decided to not commit the build directory to github.
There's no rule which required the build directory to be present at github although ist usual provided. Without build directory your adapter cannot be installed directly from github. So its mandatory to add the flag
 "nogit": true,

to io-package.json.
see https://github.com/ioBroker/ioBroker.admin/blob/1069d2e0514f2a80b4aeeb13806ed9190b0372c4/io-package.json#L184 for an example.
PLEASE ADD AS SOON AS POSSIBLE - a new release is not required as the repositories are reading the info directly from github.

Done


[TwigComponent] Feature request: boolean attributes in html syntax

I've got a Button component with some boolean properties. Let's say I have public bool $set = false added to it. To set that to true I have to use one of these methods in the html syntax:
<twig:Button set="1" />
<twig:Button :set="true" />

I would expect to also be able to do this:
<twig:Button set />

Would that even be possible? :)

I just tried and... it works already :)
<?php
// ...

#[AsTwigComponent]
final class FooBar
{
    public bool $set = false;

    public bool $isFoo = false;

    public bool $withBar = false;

    public bool $bazzed = false;
}

{# in base.html.twig #}
       <twig:FooBar
            set=""
            enabled
            isFoo="false"
            withBar
            bazzed
       />

// dump() inside Foobar.html.twig
  "set" => false
 "isFoo" => false
 "withBar" => true
 "bazzed" => true

Is there something more you'd want ? Or maybe something is not working as expected ?

I does not work with anonymous component

This issue was explicitely about class-based component, but let's ignore that :)
The same test with an anonymous component
<twig:FooBar
      set=""
      enabled
      isFoo="0"
      withBar
      bazzed
 />

when dumped from inside the component, renders like this...

  "set" => ""
  "enabled" => true
  "isFoo" => "0"
  "withBar" => true
  "bazzed" => true

I'm not sure to see where is the problem, or i may have missunderstood something (more than possible) ?

"set" => ""
"enabled" => true
"isFoo" => "0"
"withBar" => true
"bazzed" => true

so isFoo should be false isn't it? :)

Why ? If you pass "0" to an anonymous component.. why should it be casted to false ?

Why ? If you pass "0" to an anonymous component.. why should it be casted to false ?

Oops,misread

Maybe we need to add a better way to cast props in a anonymous component:
{% props label = 'Click' as 'string' ? WDYT?

Yes, it seems to work! I have no idea why it didn't seem work for me before, so sorry for the confusion...

@DemonTPx can be closed then?


Keep VRAM usage and faster slicing consistent in attention.py

EDIT:
Refactor attention.CrossAttention to remove 4 copies of einsum_op_compvis code.
Remove zeros tensor that's useless when we have enough VRAM.
For cuda keep VRAM usage and faster slicing consistent.

FYI @Any-Winter-4079 since you edited this recently.

I have tested thei PR on MacOs
512512. -s50  /  OK but there is not improvment in speed
896576 -s50 / OK but there is not improvment in speed
1024*1024 -s50 / FAIL with error Error: product of dimension sizes > 2**31'

Yeah, this is going to be tricky :)
For ongoing development of the solution, I would seek a thumbs up from 8GB (@Vargol), 16GB (@netsvetaev, @krummrey), 32GB (@0t0m0, @jroxendal) and 64 (@Any-Winter-4079) or 128GB (@i3oc9i) M1 machines before merging, as they seem to report various results. It should maintain the same speed for 512x512 and not crash on larger images.
I do agree there is probably a unified way to do this, though. I'm trying to get Textual Inversion working on M1, but I'll try to take a look. Thanks!

Going to be a while before I can get around to testing, I notice a lot of the VRAM reduction fixes have been remove which concerns me, but I guess the move to functions means they should fall out of scope and get collected so it might not be a disaster.

@Any-Winter-4079
Thumbs down from me :(
The 512x512 image was created without problems in the normal time.
However, the 1024x1024 image took half the expected time and was colorful noise afterwards.
dream> banana sushi -W512 -H512 -Ak_lms -S50
100%|███████████████████████████████████████████| 50/50 [00:27<00:00,  1.83it/s]
Generating: 100%|█████████████████████████████████| 1/1 [00:31<00:00, 31.24s/it]
>> Usage stats:
>>   1 image(s) generated in 31.37s
Outputs:
[2] outputs/img-samples/000115.50.png: "banana sushi" -s50 -W512 -H512 -C7.5 -Ak_lms -S50

dream> banana sushi -W1024 -H1024 -Ak_lms -S50
>> This input is larger than your defaults. If you run out of memory, please use a smaller image.
100%|███████████████████████████████████████████| 50/50 [07:34<00:00,  9.09s/it]
Generating: 100%|████████████████████████████████| 1/1 [07:52<00:00, 472.54s/it]
>> Usage stats:
>>   1 image(s) generated in 472.67s
Outputs:
[3] outputs/img-samples/000116.50.png: "banana sushi" -s50 -W1024 -H1024 -C7.5 -Ak_lms -S50

Non-scientific findings (i.e I shut down some of the worst memory hogs but didn't do a full restart to clear up memory)
32GB m1 max:
512x512 same as development, so about 35s.
896x576 hovering around 20s/it.
1024x1024 suspiciously fast at around 5s/it and generated colourful noise.

Sorry, can't test, seem to be in dependancy hell, I think something has been upgraded to sue a newer version of protobuff
that incompatible with a bunch of other dependancies

For ongoing development of the solution, I would seek a thumbs up from 8GB (@Vargol), 16GB (@netsvetaev, @krummrey), 32GB (@0t0m0, @jroxendal) and 64 (@Any-Winter-4079) or 128GB (@i3oc9i) M1 machines before merging, as they seem to report various results. It should maintain the same speed for 512x512 and larger images (e.g. 1024x1024) and not crash on the latter.

My problems with ram may be related to non-arm brew and installation process differences. So I need to reinstall it. Right now fresh install do cause some errors though.

After a fresh install I've got 1.38s/it 512, 6.20s/it 768 and 20.5s/it 1024 (main branch). So it were mine problems with something non-arm.

It should maintain the same speed for 512x512 and larger images (e.g. 1024x1024) and not crash on the latter.

Development:
512 at 1.04it/s (less than a second per it). 50s total.
768 at 6.20. 1024 at 22-24s/it.
This attention.py version:
512 at 1.04s/it (slower)
768 at 10s/it
1024 had crashed dream.py with this:
/AppleInternal/Library/BuildRoots/5381bdfb-27e8-11ed-bdc1-96898e02b808/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:705: failed assertion `[MPSTemporaryNDArray initWithDevice:descriptor:] Error: product of dimension sizes > 2**31'
Abort trap: 6
/Users/artur/miniconda3/envs/ldm10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Tag me when this is ready for review. You might want to convert this PR into a draft if you're still actively working on it.

Sorted my dependency hell.
512x512, around normal speed
1024x1024 was running little slow (67s/it cf 55s/it) then crashed in 35th sample
Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)

Thank you for testing this!
I see there's an issue with 1024x1024 / large sizes product of dimension sizes > 2**31 for mps (note 1024x1024 runs fine on just cpu) and I'll address that (looks like it was handled through slice_size = math.floor(2**30 / (q.shape[0] * q.shape[1]) before my change).
@netsvetaev Surprised about any slowdown. This change is mostly a refactoring, especially for 512x512 where it should trigger the same algorithm with no slicing. I read you have 16GB, so for 768x768 size it should use larger slices. I'm guessing larger slices are slower, which is okay as I have to limit them to fix the above anyway.

@netsvetaev I'm in the office right now, will try it later this evening.

Darn, just realised that means I'm getting those memory issues from main, wasn't getting any from the 1.14.3 test repo.

@mh-dm
after force-pushed the attention branch from 297e70d to b93338a
banana split -s50 -W1024 -H1024  works but it is almost two times slower than 1.14.1 release (2m55s)
50/50 [05:43<00:00,  6.87s/it]
1 image(s) generated in 345.72s

banana split -s50 -W576 -H896 works and it s just a bit slower (50s in 1.14.1 relese)
50/50 [00:54<00:00,  1.09s/it]
1 image(s) generated in 55.06s

banana split -s50 -W512 -H512 works at the same speed of 1.14.1 relese
50/50 [00:22<00:00,  2.25it/s]
1 image(s) generated in 22.83s

On, I can confirm in now for me 512x512 is 400% slower

There is not lscpu in MacOs M1 sorry

Can someone give me a hand, how do I check out a pull request?
git pull origin pull/582/head:development didn't work.

Maybe just git pull origin pull/582/head?

I ran it 3 time each (-n3):
512x512  -  1:02, 1:04, 1:04
640x640  -  3:33, 3:39, 3:27
No improvement here, even a slight increase in render times.

Thank you for the test. Were you on b93338a or ac32c03 ?

One small off by one error fixed and using smaller max tensor sizes by default. I think smaller sizes help fit some operations in L2/L3 caches which is why there was a perf difference before, or at least that's my best theory from the numbers you've kindly provided. If I limit to 32MiB then I get an almost 20% speed increase on intel ~10s to ~8s/it.
Can anyone on M1 give it a try?
Current max tensor size limit logic for mps, could be easily tuned up or down based on your testing:
        # mps have larger caches. But don't go too high to avoid:
        # 'large sizes product of dimension sizes > 2**31' error on mps
        if self.mem_total_gb <= 8:
            return self.einsum_op_tensor_mem(q, k, v, 64)
        elif self.mem_total_gb <= 16:
            return self.einsum_op_tensor_mem(q, k, v, 128)
        else:
            return self.einsum_op_tensor_mem(q, k, v, 256)

@mh-dm sysctl -a displays a lot of info on Mac.
For M1 Max 64GB
sysctl -a | grep hw.l1
hw.l1icachesize: 131072
hw.l1dcachesize: 65536

Those are L1 instruction and data cache sizes
sysctl -a | grep hw.l2
hw.l2cachesize: 4194304

It doesn't appear to return any info on L3, so maybe there's not one (?).

Can someone give me a hand, how do I check out a pull request?
git pull origin pull/582/head:development didn't work.

Checkout a pull request
% git fetch origin pull/582/head:test_582
% git checkout test_582
Thats what I've been using

Checked it out the Vargol way and ran it 3 time each (-n3):
512x512 - 1:03, 1:04, 1:03
640x640 - 3:19, 3:47, 3:23

Ok, really promising results from latest commits:
512x512 same as usual.
1024x384 1m14s, about 15seconds faster than lstein:development
1024x1024 took about 9 minutes, but this is the first time I'm completed generation at that size! no visible artefacts. memory pressure didn't even hit the ceiling, I think I could go every higher. Very promising indeed.

@jroxendal
Whats your set up, been doing 1024x1024 on a 8GB shared memory Apple M1 for a while, although the release version of 1.14 seems to have issues the 1.14.3 performance test version didn't in that regard

Still 400% slower here at 512x512

@jroxendal Thank you. First speedup confirmation 🎉, on 32GB M1 max. From code that should be limiting tensor size to 256MiB
return self.einsum_op_tensor_mem(q, k, v, 256)

@krummrey 16GB M1 basic right? I don't have baseline (git checkout origin development) numbers from you, so I can't tell whether slower or faster. But the high variability for 640x640 tells me to go down from 128 to 64MiB max tensor size.

Still 400% slower here at 512x512 and use 2 -3 more GB than the release version.
@Vargol This is surprising given the code switch to 64MiB max tensor size for <=8GiB. The higher ram usage is indicative of using an older version (the one with 1024), maybe a quick refetch & checkout and make sure you're on commit 2df118d? You could also try replacing the whole einsum_op_cpu like

def einsum_op_cpu(self, q, k, v):
  return self.einsum_op_tensor_mem(q, k, v, 64)

And try different values like 64, 32, 24, 16.

@Any-Winter-4079 Thank you for that info. sysctl -a doesn't show L* cache sizes on linux unfortunately so we're still missing a common way of getting the values. Could you give the latest code, 2df118d, in this PR a try on 512x512?
Has anyone tried getconf -a | grep CACHE_SIZE?

@mh-dm I'm afraid I have to retract my speedup report from before:
development branch 1024x1024 is at ~8s/it and this branch at 12s/it, so slower. Pardon the confusion.

@jroxendal Thanks. How's 512x512 and some size in between like 640x640?

@Any-Winter-4079 Thank you for that info. sysctl -a doesn't show L* cache sizes on linux unfortunately so we're still missing a common way of getting the values. Could you give the latest code, 2df118d, in this PR a try on 512x512 and maybe 640/768?
Has anyone tried getconf -a | grep CACHE_SIZE?

getconf -a | grep CACHE_SIZE doesn't work on M1, but you can always get the data in different ways based on the device type, e.g. if torch.cuda.is_available(): do X else do Y

512x512 [00:33<00:00,  1.47it/s] dev
512x512  [00:35<00:00,  1.40it/s] this branch
640x640 [01:01<00:00,  1.23s/it] dev
640x640 [01:17<00:00,  1.55s/it] this branch

@Any-Winter-4079 Thank you. I know I can switch on device type, but I was looking to switch on more granular information. If L3_cache_size_mb <= 8: split at 32 else split at 64

You'll need to take into account that your CPU ops methods are currently dealing with M1 'APUs' and intel CPU with AMD graphics cards, they all fall under the 'mps' device type as its METAL based not M1 specific.

@jroxendal Thank you.
Okay, time to pivot. I will undo and undo parts of this change until it will be a pure refactoring for mps.

I've brought back einsum_op_mps_v1 and einsum_op_mps_v2 and the exact same logic to pick between the two. Also rebased on top development head. Could anyone test on M1* to confirm that speeds for say 512x512 and 640x640 are unchanged compared to development head?

@Any-Winter-4079  beat me to it, currently testing with the >= fixed to > than as without the fix 1024x1024 is  4 MINUTES per it

@Any-Winter-4079 Not sure that typo matters in practice as RAM nowadays isn't exactly integer GiBs so it won't divide equally. Regardless the issue is in main and this PR no longer changes that.

@Any-Winter-4079 Not sure that typo matters in practice as RAM nowadays isn't exactly integer GiBs so it won't divide equally. Regardless the issue is in development head and this PR no longer changes that.

Yes it matters ... Some one with a 8Gb M1 :-)
As your code has that line in too so if I was to test this PR as is ir would have a great big thumbs from me.

Yes it matters ... Some one with a 8Gb M1 :-) Your code has that line in too so if I was to test this PR 'as is' it would have a great big thumbs down from me.

8GB is different from 8GiB which is probably different from what psutil.virtual_memory().total will report if size isn't exact or some memory is reserved. For example, I have 32GB memory, but:
>>> import psutil; print(psutil.virtual_memory().total / (1024**3))
31.121078491210938

Try it as an easy way to check if the = in self.mem_total >= 8 makes a difference or not.

I don't need to try it

= 8      512x512   @ 25 s/it
8        512x512   @   5 s/it

Thanks for checking. Are the numbers for the latest version of this pull request?

As long as you haven't updated it since
https://github.com/lstein/stable-diffusion/pull/582#issuecomment-1249290586
then yes its with the latest

Great, then including switch to self.mem_total > 8  in this PR.

Yes it matters ... Some one with a 8Gb M1 :-) Your code has that line in too so if I was to test this PR 'as is' it would have a great big thumbs down from me.

8GB is different from 8GiB which is probably different from what psutil.virtual_memory().total will report if size isn't exact or some memory is reserved. For example, I have 32GB memory, but:
>>> import psutil; print(psutil.virtual_memory().total / (1024**3))
31.121078491210938

Try it as an easy way to check if the = in self.mem_total >= 8 makes a difference or not.

Interesting. That is probably a difference with Macs. Here it's exact values, so 64.0 means 64GB Mac and so on.
>>> psutil.virtual_memory().total
68719476736
>>> psutil.virtual_memory().total / (1024**3)
64.0

Yep, it's strange that someone decided to start applying fake disk storage units to memory as while, I'm sure it saves someone a penny a memory stick or something.

test on commit bcfa635aef35b55d902c43d6b8302db79f009513
"banana split" -s50 -W1024 -H1024 -C7.5 -Ak_lms
[02:52<00:00, 172.18s/it]
1 image(s) generated in 172.37s

just 3 sec faster than last development branch 45af30f3a4c98b50c755717831c5fff75a3a8b43
FAIL because I get noise (did two runs to cofirm)
"banana split" -s50  -W896 -H576 -C7.5 -Ak_lms
50/50 [00:53<00:00,  1.07s/it]
1 image(s) generated in 54.10s

OK same execution speed
"banana split" -s50  -W512 -H512 -C7.5 -Ak_lms
50/50 [00:21<00:00,  2.31it/s]
1 image(s) generated in 22.19s

OK same execution speed

512x512 ... 👍
100%|████████████████| 50/50 [03:51<00:00,  4.63s/it]
100%|████████████████| 50/50 [04:00<00:00,  4.80s/it]
100%|████████████████| 50/50 [04:02<00:00,  4.85s/it]
Generating: 100%|████████████████████████████████████| 3/3 [12:28<00:00, 249.49s/it]

Usage stats:
3 image(s) generated in 748.69s

1024x1024 looking good speed wise (despite it throwing out an error everyone again, nothing to do with this change though)
but it'll be 2.5 hour before I can confirm cos thats how slow it is for me :-)

@i3oc9i Thank you. Hmm.. I'm assuming you don't get noise at dev head for 1024. I did another self review and checked for any logic einsum_op_mps_v1 in einsum_op_mps_v2 that might result in any difference. One I see is s = einsum('b i d, b j d -> b i j', q, k) * self.scale vs s = einsum('b i d, b j d -> b i j', q, k); s *= self.scale. That would be weird if it were the cause, but that gave me an idea on how to get a clear speedup: moving * self.scale to before einsum(), specifically to k that's most often both smaller than q and always much smaller than s.
Would appreciate further testing.

that actually might be it, I can't test it of course as I never hit the CompVis op, however it looks similar to this commit that was done to back out issues when training
https://github.com/lstein/stable-diffusion/commit/bf1beaa607984e34f3bfab8b264444318b6f47b0
In that case a bunch of += thad to be reverted back to addition.

@mh-dm
test on commit 1114da7b072e236c6f84deb00923485699e9fe0b
"banana split" -s50 -W512 -H512 -C7.5 -Ak_lms. (OK same speed)
"banana split" -s50 -W896 -H576 -C7.5 -Ak_lms (OK a bit faster)
"banana split" -s50 -W1024 -H1024 -C7.5 -Ak_lms (OK 2:55 😀 You got it 👏)

@i3oc9i Awesome 🎉!
I think I can call it ready for review :)

    def einsum_op_mps_v2(self, q, k, v):
        if self.mem_total > 8 and q.shape[1] <= 4096:
            return self.einsum_op_compvis(q, k, v)
        else:
            return self.einsum_op_slice_0(q, k, v, 1)  <-- ****I always go here****

Yeah and einsum_op_slice_0 calls einsum_op_compvis for each slice.

Guys, I'm on 16gb m1.
Right now on main I have 1.3/512 at minimum, but 5.8-6.2s/768, 19s/1024 (~15m total);
on dev 1.04it/512 (+30% faster), but 6.7s/768 (slower), 26s/1024 (slower, 21m total).
P.S. «q» doesn't work on dev.

Yeah and einsum_op_slice_0 calls einsum_op_compvis for each slice.

Then I apologise,I must have been  remembering the code wrong

@netsvetaev Could you please compare this PR to development? Not sure what «q» is?

Sorry, I thought this is the development. How I can compare then?
«q» — command for quit dream.py.

git fetch origin pull/582/head:test_582
git checkout test_582

Now you're on this PR and you can test.
git checkout HEAD^

Now you're on development (since this PR is only one commit) at whatever this PR was last synced to.
If you just want latest development:
git checkout origin/development

@netsvetaev Could you please compare this PR to development?

1.05it/512
6.23s/768 (jumps from 5.7 to 6.7)
23.5s/1024

"banana sushi" -s50 -C7.5 -n3 -W512 -H512
3 image(s) generated in 81.13s (86s previously)
"banana sushi" -s50 -C7.5 -n3 -W1024 -H1024
3 image(s) generated in 716.92s (782s previously)
Big thumbs up from me @mh-dm . Having debug prints and other stuff running, and even then, best times.

Do we have the thumbs up from everyone from 8 to 64/128GB?
64 and 128GB is a 👍

Do we have the thumbs up from everyone from 8 to 64/128GB?
64 and 128GB is a 👍

I see some slowdown when there is not enough memory. Maybe it’s a random case. 512 seems faster a bit, 768 more or less the same, 1024 somewhat sower. It would be great if there was a way to deal with it.

I ran it 3 time each (-n3):
512x512 - 1:14, 1:06, 1:12
640x640 - 3:03, 2:45, 2:39
512x512 went right into swap which it didn't do before, I guess thats where the increased rendering time comes from. (when I tested v1.14 I had some renders under a minute. so 10-15% faster back then)
640x640 is faster now.

I ran a proper 1024x1024 overnight and the results where in the 60 - 66 s/it region, which is in the right area and I get very inconstant values at that size anyway, probably use to OS things kicking in and stealing bits of memory.
512x512 is as fast as usual
-W704 -H512 was slow 1709 vs 1299 for my best run, I need to test against development to see what happen, but thats not going to happen for a few hours.

"banana sushi" -s50 -C7.5 -n3 -W704 -H512 is faster for me (64GB)
3 image(s) generated in 124.35s (133s before)
Just to report at some RAM sizes, the current code is faster at that size too.

It's going to be really hard for me to deliver mps specific optimizations, as somewhat experienced in this thread :). Best I can do is look for architecture independent optimizations (like moving * self.scale earlier on a smaller tensor) and maintain existing architecture specific optimizations while refactoring, like how I've brought back einsum_op_mps_v1 and einsum_op_mps_v2 and the logic to switch between them.
Extra difficulty comes from variability while timing/profiling and external factors. Just an example, I've found out that which power supply I use significantly affects my results on cpu - 12s/it to 10s/it (timed both on development). That's on top of known other factors like: amount free RAM, any other apps running, whether plugged in or not and performance profiles.
So, I've updated the code to more cleanly separate the architectures, specifically to separate paths for mps and cpu (where I could now test and apply a smaller&faster max tensor size). I've also rebased to top of development head. With everything refactored it should be easier for someone on mps to go in and try out future optimizations. Maybe it works to switch to self.mem_total > 16 in einsum_op_mps_v2 or maybe to just call einsum_op_tensor_mem(self, q, k, v, some_ideal_limit_mb) or maybe something else.
With all the testing (thank you!) and many good numbers I'm hoping we can call it good enough soon. Also, would appreciate some fresh eyes on the code looking for any bugs left.

@mh-dm Let me say I appreciate your efforts. You are clearly more knowledgeable than I am, but still it seems this issue is a pain in the ass :)
The optimization works great for 64GB and 128GB: For me at 64GB, it's 10% less time al 1024x1024 and 6% less time on 512x512 so that's pretty significant for me! 👍  If 16-32GB report worse results, we can always let them use the current dev version (meaning, just route them to the existing function -which is very simple to do, because reported RAM on Mac is exact.
So here self.einsum_op = self.einsum_op_mps_v1 if self.mem_total >= 32 else self.einsum_op_mps_v2 we can route them to the existing version if they prefer so. Anyway, thanks again!

on last commit 654ab4eb5d2ad7201d3ebf091810f51de459a97a
"banana split" -s50 -W512 -H512 -C7.5 -Ak_lms. (OK 0:21)
"banana split" -s50 -W896 -H576 -C7.5 -Ak_lms (OK 0:50 )
"banana split" -s50 -W1024 -H1024 -C7.5 -Ak_lms (OK 2:44)

"we can always let them use the current dev version (meaning, just route them to the existing function -which is very simple to do, because reported RAM on Mac is exact)"
If you look at the code, that's basically what it's doing. @Vargol reported okay for 512x512 on 8GB and for that specifically I did the >= 8 to > 8 change requested. It's only @krummrey on 16GB that's reporting a slowdown. That slowdown I don't understand at all because all the splitting logic is the same. However, netsvetaev@ also on 16GB reports "512 seems faster a bit, 768 more or less the same, 1024 somewhat slower". I believe further optimizations/changes specific to 16GB should be in a future change worked on by someone with access to said machine.

I'm still around an hour  from confirming for definite but my slow down at
-W704 -H512  seems to be inherent to the dev version, still need to check main
For my old fork, 100 odd commits behind main

Usage stats:
3 image(s) generated in 1244.17s

For the development branch, checked out and pulled to make sure its up to date

Usage stats:
3 image(s) generated in 1629.39

The development branch, replacing attention.py and model.py with copies from my fork...
Not finished yet but the first iter was 10.5s/it very similar to the dev branch, the
second is 50% though at also showing ~10.5s/it

@Vargol You're getting a "400%" speed up for 512x512 from this PR, i.e. the change from >= 8 to > 8 :D, compared to development. Don't think there's more this PR can do to top that :).

I re-opened review but I forgot to re-tag @lstein

If you compare 64GB vs. 128GB:
"banana split" -s50 -W512 -H512 -C7.5 -Ak_lms. (OK 0:26)
Comparison with 2x RAM: 24% slower
"banana split" -s50 -W896 -H576 -C7.5 -Ak_lms (OK 1:10)
Comparison with 2x RAM: 40% slower
"banana split" -s50 -W1024 -H1024 -C7.5 -Ak_lms (OK 3:44)
Comparison with 2x RAM: 37% slower
If you compare (best from) 16GB vs. 128GB:
"banana split" -s50 -W512 -H512 -C7.5 -Ak_lms. (OK 0:48)
Comparison with 8x RAM: 85% slower
"banana split" -s50 -W896 -H576 -C7.5 -Ak_lms (OK 5:12)
Comparison with 8x RAM: 624% slower
"banana split" -s50 -W1024 -H1024 -C7.5 -Ak_lms (OK 19:35)
Comparison with 8x RAM: 716% slower
Clearly, it's not just GPU cores that matter, but there's probably swap going on. So in the future, to optimise 16GB even further, one would have to probably look at that. But without a 16GB Mac like @mh-dm said, it's hard to optimise.

Yes, ignoring that the the dev branch was a 400% slow down from main due to that typo:-)
But the point of that post was to prove to myself, even if no one else, that the issues I'm seeing with '-W704 -H512 ' is nothing to do with this PR.

@mh-dm Yep, you were right that it is almost the exactly same code for 16GB.
This returns your memory, e.g. 16 for 16GB.

With 16GB, you'd go through einsum_op_mps_v2

For images 512x512, run einsum_op_compvis

which is

Else, use einsum_op_slice_0 with slice_size 1

Compared to current dev branch code (with >=8 typo)

Maybe the available RAM at any point affects performance (e.g. one day may report faster results than a different day, because of other apps, etc. running).
If the different results from dev branch were from the same day, I'm not sure what it could be. Maybe calling the einsum_op_compvis  function with portions of the data, e.g. r[i:end] = self.einsum_op_compvis(q[i:end], k[i:end], v[i:end])  (which I don't know how it works -maybe the slice of data is copied using ranges?)
Other than that, I really don't see any difference.

Thanks for looking. Can I get the github review approve so I can submit then?

Thanks for looking. Can I get the github review approve so I can submit then?
Oh and to answer the question, slices like q[i:end] are not copies.

By the way, have you tested on CUDA?

Yeah, tested ~6% faster on 8gb nvidia card for 512x512. I tested various sizes until 1024x1024 at which point it was ~10% faster. Intel v7 cpu got the highest improvement, I think ~25% overall for 512x512 down to 7-8s/it though.

@krummrey @netsvetaev
By the way, because 16GB have reported mixed results, if anyone with 16GB RAM tests (same time of day, same apps open/closed, exiting dream> after every test, leaving a few minutes not to overheat -which is a real thing for me at 64GB impacting performance) and there seems to be a consistent slowdown, you can always do a Pull request to add your own code to run for 16GB Mac. For example here

Just need to add an if, e.g. if self.mem_total_gb==16 and then just call the current code you are running (or an even more improved version). The current code (before this merge) is explained here. https://github.com/lstein/stable-diffusion/pull/582#issuecomment-1250106383 👍

One thing I'd try when experimenting with optimizations is changing > 8 into > 16 to always call slice_0.
Another thing to try would be:
    def einsum_op_mps_v2(self, q, k, v):
        if self.mem_total_gb <= 8:
            return self.einsum_op_slice_0(q, k, v, 1)
        return self.einsum_op_tensor_mem(q, k, v, 64)

Where you try different values, 64, 48, 32, 24, 16.

Seems it's got faster on a latest dev:
1.11it/s 512 (was 1.04 maximum), 45s total
5.25s/it 768 (was 6.20), 04:22s total
That's impressive!
For 1024 I have same numbers from 23s, increasing with every sample up to 25-26 (was 19s as best result earlier).
I will try to edit the code, thank you for the tips.

on last commit e040c35d34131c7ffcc55bd9c1ec8e21d5f48025
"banana split" -s50 -W512 -H512 -C7.5 -Ak_lms. (OK 0:20) same as previous report
"banana split" -s50 -W896 -H576 -C7.5 -Ak_lms (OK 0:50 )  same as previous report
"banana split" -s50 -W1024 -H1024 -C7.5 -Ak_lms (OK 2:44) same as previous report

I've tried this out on my fork:
https://github.com/Birch-san/stable-diffusion/compare/birch-mps-waifu...optimized-attention
using very recent PyTorch nightly: 1.13.0.dev20220917 on M1 Max (64GB).
1 sample:
optimized:
11.4 secs
non-optimized:
10.4 secs
3 samples:
optimized:
37.8 secs
non-optimized:
30.2 secs
Conclusion
Optimized attention is still slower than "CompVis original code + some dels". for small batches and also large.

@Birch-san  Have you tried in with a release pytorch I see significant slowdown with the nightlies.

using PyTorch stable 1.12.1
1 sample
optimized:
10.1 secs
non-optimized:
9.8 secs
3 samples
optimized:
31.2 secs
non-optimized:
28.7 secs
Conclusion
Optimized attention is 3~9% slower than "CompVis original code + some dels".
PyTorch stable is 5% faster than recent nightly.

interesting, it seems I'm being sent down the M1 16–32GB code path. my memory_available was measured to be 31.89 GB. 😛

using PyTorch stable 1.12.1
1 sample
optimized:
9.8 secs
non-optimized:
9.8 secs
3 samples
optimized:
26.9 secs
non-optimized:
28.7 secs
Conclusion
If you go down the right code path (i.e. quitting apps to free up 32GB of memory), then optimized attention is identical speed for 1 sample, and 6.7% faster for a batch-of-3 samples.

btw, I think I see a way to make inference faster.
instead of allocating new tensors each time: we could retain and re-use tensors.
it'd be a bit of a mess though. potentially increases the peak VRAM use (because you'd have a few large tensors that you'd never free).
I also wonder whether slapping .contiguous() on any of these tensors would make it cheaper to do arithmetic on them. due to improved locality.

btw, I think I see a way to make inference faster. instead of allocating new tensors each time: we could retain and re-use tensors. it'd be a bit of a mess though. potentially increases the peak VRAM use (because you'd have a few large tensors that you'd never free).

pytorch really doesn't like modifying/re-using tensors. I had to re-land #569 with a .clone() fix so that training still works and even a simple *=  scalar seemed to introduce issues for 1024x1024 on M1. Finally, there's not that much left to speed up unfortunately when the slowest operation/the bottleneck is s = s.softmax(dim=-1) for which there's no inplace version available in pytorch. If anyone reading has contributed to pytorch before and would like a nice rewarding challenge try adding support for softmax_ or softmax(inplace=True).

I also wonder whether slapping .contiguous() on any of these tensors would make it cheaper to do arithmetic on them. due to improved locality.

Less chance of seeing significant improvements as you'd be introducing a copy. Plus I think reshape() already results in contiguous copy.

interesting, it seems I'm being sent down the M1 16–32GB code path. my memory_available was measured to be 31.89 GB

You have the configuration to try out M1 16GB+ specific optimizations that Any-Winter-4079 and I have suggested, just a few messages up.

interesting, it seems I'm being sent down the M1 16–32GB code path. my memory_available was measured to be 31.89 GB. 😛

If you go down the right code path (i.e. quitting apps to free up 32GB of memory), then optimized attention is identical speed for 1 sample, and 6.7% faster for a batch-of-3 samples.

@Birch-san This new commit doesn't use mem_available on M1 though. It uses mem_total. You shouldn't need to quit apps nor get 31.89 of available memory (b/c it's mem_total that's measured).


flash player

is there way to use flashplayer by using pep-flash inside this version?

This suggestion is not feasible with newer chromium versions and thus not with jcefmaven. Read more on it here: https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=15959&p=47204


HIVE-25378 ci merge

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

xxx


feat: Add Parcel 2 Support for Million.js

Describe the feature
I'm a big fan of Million.js and its overall concept, which is truly wonderful. However, I'm currently working on a project that uses Parcel 2 as the bundler/compiler, and I've encountered limitations when trying to integrate Million.js with Parcel 2.
While I've managed to get Million.js partially working with Parcel 2, the limitations stem from not being able to use the Million.js compiler. This means I cannot take full advantage of Million.js's advanced features. I'd love it if some support could be added.
Why should this be a priority
While parcel2 in itself is not too popular, the very popular extension framework plasmo is built on top of parcel. Meaning that if you add support for parcel2, then we get support for plasmo and vice-versa.
Additional information

[X] Would you be willing to help implement this feature?

I have the exact same use case. I am developing an extension with plasmo with react, but would love to incorporate million into it.

I'm also missing out on integrating Million for my Chrome extension built with Parcel.

I'm humbly begging for the Parcel crew to get some love @aidenybai 🙏🏽 I spent all day trying to get multiple vite ports working just to try Million.js out in my excitement (they all failed).


Fix/390

This PR implements issue #390 by adding the two default date ranges as well as adding a form for picking custom start and end dates. Implementation is as image attached.

Thanks Sunday!  I will get some feedback together for you.

So I was thinking about the way the date range filter looks and works, and think we would be better  to use this: https://uxsolutions.github.io/bootstrap-datepicker/?markup=component&format=&weekStart=&startDate=&endDate=&startView=0&minViewMode=0&maxViewMode=4&todayBtn=false&clearBtn=false&language=en&orientation=auto&multidate=&multidateSeparator=&keyboardNavigation=on&forceParse=on#sandbox
Can you also left align the start and end labels on with the same left margin that the other options have?

Okay, looking into that now.

this has been updated now, and rendering as follow:


Remove redundant logic

Depends on https://github.com/ros2/rosidl/pull/432

Now that rosidl_generator_c is released, I've opened a follow-up adding the minimum version dependency: https://github.com/ros2/rosidl_typesupport_connext/pull/54


Map editor generates invalid syntax with sprites with backtick key

Describe the bug
When a sprite with a backtick (`) as its key is placed using the map editor, the editor places the backtick unescaped, ending the string literal causing invalid syntax which is not obvious until closing the map editor.
To Reproduce
Steps to reproduce the behavior:

Add a sprite with a backtick as a key to the legend
Add the sprite to a map in the map editor

Expected behavior
The sprite is placed on the map without breaking syntax, possibly through escaping the backtick.
Actual behavior
The backtick is placed in string and terminates the string early.
Screenshots
https://github.com/user-attachments/assets/efd36695-f586-497f-a4b7-77726b619ff2
Since escaping characters means that 1 character may not always = 1 tile, it might be better to just prevent using backtick for a sprite key.
(A separate issue where the map clicked on to edit is not the one that ends up being edited is also exhibited in the video but is not the subject of this issue.)

I think the editor should warn users against using backticks as sprite keys.


Allow the configuration of filesystem permissions.

Hello! I've been setting up rain in k8s and serving downloaded files locally using a samba server but I ran into the issue that rain creates files and directories with file permissions 0750 and 0640 (which the samba doesn't export). The easiest solution for me was to have rain create files as 0755&0644.
This PR adds the config option FilePermissions allowing the users to choose which permissions files and directories should be created with. For directories it takes the config value and for files it'll strip off the execution bit of the value (that's a bad idea in any case). e.g if you set it to 0755 files, will be created with 0644.
Hope this can be of help. 👍🏻

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

3 of 7 (42.86%)  changed or added relevant lines in 4 files are covered.
6 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.08%) to 45.32%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

torrent/session_load.go
0
1
0.0%

torrent/session_rpc_handler.go
0
3
0.0%

Files with Coverage Reduction
New Missed Lines
%

torrent/torrent_messagehandler.go
6
42.86%

Totals

Change from base Build<PHONE_NUMBER>:
-0.08%

Covered Lines:
3336

Relevant Lines:
7361

💛  - Coveralls

Thanks @koenbollen . I have released a new version v1.10.0.

That was quick! ✨ Thanks! 👍🏻


Implement SearchSuggestions component

PR for Task #7 to replace PR #25

@reynard2007 I think after recent PR merges, it again got conflicts. Can you please resolve them?

Closing. To be replaced by PR #34


Pipeline Testing

Addresses #1148
This tests the current capability of pipelines. I've marked a few areas of improvements in the comments and will create separate issues for potential upgrades.

@zacandcheese can you please make the changes requested by Umar so we can close this PR.

@mail4umar please check again Zac PR and see if the changes are what we need.


WIP- add timezone dropdown to reports

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

This was separated into smaller PRs


Bump sundr-builder-annotations from 0.101.0 to 0.101.3

Type of change

Enhancement / new feature

Description
We discovered a regression in the 0.101.2 release (#698).  The project's now put out 0.101.3 in response. The issue has been fixed.
This is a build time test dependency, so no changelog entry is applicable.
Additional Context
Why are you making this pull request?
Checklist
Please go through this checklist and make sure all applicable tasks have been done

[ ] Write tests
[ ] Make sure all tests pass
[ ] Review performance test results. Ensure that any degradations to performance numbers are understood and justified.
[ ] Make sure all Sonar-Lint warnings are addressed or are justifiably ignored.
[ ] Update documentation
[ ] Reference relevant issue(s) and close them after merging
[ ] For user facing changes, update CHANGELOG.md (remember to include changes affecting the API of the test artefacts too).

@SamBarker could you review this one, merge at will.


Update README.md example

Main example has multiple missing features which cause it to fail. This PR aims to resolve those that I found.

I have read the CLA Document and I hereby sign the CLA


Add steps to the time slider

I think the version where we only have the single time slider, would benefit greatly from this feature.
A steps property with a value of 5 would mean that only the 5 minute intervals can be selected (5, 10, 15,...)
As also described in https://github.com/wangzuo/input-moment/issues/64

Which time sliders? All of them? It shouldn't be that hard to implement. It would depend on the underlying slider framework if it has that option. Open to pull requests.

Is it react-input-slider you're using? If so, it's looks like it got support for this, using the xstep parameter.

It looks like it is indeed react-input-slider. This would be awesome.

+1

Yeah it wasn't good on mobile defaults defaulted to 5 steps.


EnumMappingStrategy not working on individual mapping.

Hi, I have tried to configure EnumMappingStrategy on individual mapping via the static and instance objects but it has no effect on the mapping.
TypeAdapterConfig<SimplePoco, SimpleDto>.NewConfig().EnumMappingStrategy(EnumMappingStrategy.ByName);
config.NewConfig<SimplePoco, SimpleDto>().EnumMappingStrategy(EnumMappingStrategy.ByName)
However it works when I set it globally, which is not the desired config I want. Is there something I am missing or is this a bug?

I'm seeing this as well.
Version 7.2 .Net Framework 4.8
using Mapster;

using Xunit;

namespace AdaptTests
{
    public class EnumMappingTests
    {
        [Fact]
        public void Doesnt_Work_Per_Type()
        {
            TypeAdapterConfig.GlobalSettings.Default
                .EnumMappingStrategy(EnumMappingStrategy.ByValue);

            TypeAdapterConfig<MyClassOne, MyClassTwo>
                .NewConfig()
                .Map(two => two.That, one => one.This)
                .EnumMappingStrategy(EnumMappingStrategy.ByName);

            TestEnumAdapt();
        }

        [Fact]
        public void Works_When_Setting_Globally()
        {
            TypeAdapterConfig.GlobalSettings.Default
                .EnumMappingStrategy(EnumMappingStrategy.ByName);

            TypeAdapterConfig<MyClassOne, MyClassTwo>
                .NewConfig()
                .Map(two => two.That, one => one.This);

            TestEnumAdapt();
        }

        private static void TestEnumAdapt()
        {
            TypeAdapterConfig.GlobalSettings.RequireExplicitMapping = true;

            var myClassOne = new MyClassOne
            {
                This = OneThing.TheMatchingOne
            };

            var myClassTwo = myClassOne.Adapt<MyClassTwo>();

            Assert.Equal(TwoThing.TheMatchingOne, myClassTwo.That);
        }

        public class MyClassOne
        {
            public OneThing This { get; set; }
        }

        public class MyClassTwo
        {
            public TwoThing That { get; set; }
        }
    }

    public enum TwoThing
    {
        TheMatchingOne = 1000,
        Nah = 2000,
        Rah = 3000
    }

    public enum OneThing
    {
        Rah = 0,
        TheMatchingOne = 21,
        Nah = 200
    }
}

bug？ i have same question

nobody？

EnumMappingStrategy works correctly. The problem is, that you cannot set it on configuration for those classes. It does not make sense, because there may be many different enum properties there and probably you do not want to map all of them by name. You have to set EnumMappingStrategy for enum types you want it to work for (and probably you want to set it for both directions). So the correct settings is:
TypeAdapterConfig<OneThing, TwoThing>
    .NewConfig()
    .EnumMappingStrategy(EnumMappingStrategy.ByName);
TypeAdapterConfig<TwoThing, OneThing>
    .NewConfig()
    .EnumMappingStrategy(EnumMappingStrategy.ByName);

@andrerav  EnumAdapter parent is PrimitiveAdapter and also do not support Custom Mapping processing.This is not a bug but rather a feature request )


make_array doesn't support mixed types with dictionaries

Describe the bug
Discovered this working on new array functions. DataFusion's make_array doesn't play nice with underlying Dictionary types. Kinda yet another issue related to https://github.com/apache/datafusion/issues/11513 imo.
Steps to reproduce
Two separate cases that are easy to recreate with existing unit test setup:
checkSparkAnswerAndOperator(df.select(array(col("_13"), col("_13"))))

produces
  org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 11.0 failed 1 times, most recent failure: Lost task 0.0 in stage 11.0 (TID 11) (<IP_ADDRESS> executor driver): org.apache.comet.CometNativeException: Invalid argument error: column types must match schema types, expected List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) but found List(Field { name: "item", data_type: Dictionary(Int32, Utf8), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }) at column index 0

Letting DataFusion infer the return type instead of specifying it results in
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 11.0 failed 1 times, most recent failure: Lost task 0.0 in stage 11.0 (TID 11) (<IP_ADDRESS> executor driver): java.lang.NullPointerException: Cannot invoke "org.apache.comet.shaded.arrow.vector.dictionary.DictionaryProvider.lookup(long)" because "dictionaryProvider" is null

which seems like an internal Comet issue? Haven't dug into this but presumably is fixable.
And doing a mixed dictionary/non-dictonary like
checkSparkAnswerAndOperator(df.select(array(col("_8"), col("_13"))))

produces
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 8.0 failed 1 times, most recent failure: Lost task 0.0 in stage 8.0 (TID 8) (<IP_ADDRESS> executor driver): org.apache.comet.CometNativeException: assertion `left == right` failed: Arrays with inconsistent types passed to MutableArrayData
  left: Utf8
 right: Dictionary(Int32, Utf8)

in datafusion/functions-nested/src/make_array.rs:231
Expected behavior
Not sure what the expected behavior or fix is. Either implement this function from scratch with better dictionary handling, or add some wrapper around invoking the UDF to flatten dictionary encoded arrays
Additional context
No response

@Kimahriman I've also been running into many issues around dictionary-encoded types. We have logic in CopyExce for unpacking them.

Does it make sense to make an Expression equivalent of CopyExec? Then for things like DataFusion scalar UDFs used directly you could just wrap it a CopyArrays expression that just unpacks dictionaries if they exist? Then you wouldn't need custom Rust code for handling it, can just be done in the Scala query serde. Downside is if you have multiple expressions in a single project you might end up unpacking the same thing multiple times.

Letting DataFusion infer the return type instead of specifying it results in
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 11.0 failed 1 times, most recent failure: Lost task 0.0 in stage 11.0 (TID 11) (<IP_ADDRESS> executor driver): java.lang.NullPointerException: Cannot invoke "org.apache.comet.shaded.arrow.vector.dictionary.DictionaryProvider.lookup(long)" because "dictionaryProvider" is null

I belive this would be resolved by forwarding the dictionaryProvider into the CometListVector similarly to what was done with the CometMapVector and CometStructVector in this PR: https://github.com/apache/datafusion-comet/pull/789/files

Not sure what the expected behavior or fix is. Either implement this function from scratch with better dictionary handling, or add some wrapper around invoking the UDF to flatten dictionary encoded arrays

To me this seems like a bug in the upstream datafusion implementation. Would it not be better to address the bug there? e.g make that implementation have correct behavior around dictionary encoded types.

Does it make sense to make an Expression equivalent of CopyExec? Then for things like DataFusion scalar UDFs used directly you could just wrap it a CopyArrays expression that just unpacks dictionaries if they exist? Then you wouldn't need custom Rust code for handling it, can just be done in the Scala query serde. Downside is if you have multiple expressions in a single project you might end up unpacking the same thing multiple times.

Seems like that would cause an unnecessary copy in the case of the make_array. To me it seems like we should just be "unpacking the dictionary as part of the data getting writing into the new array data structure. And other expressions might be able to do other optimizations by having data in the dictionary format for example https://github.com/apache/datafusion-comet/issues/504

I belive this would be resolved by forwarding the dictionaryProvider into the CometListVector similarly to what was done with the CometMapVector and CometStructVector in this PR: https://github.com/apache/datafusion-comet/pull/789/files

I figured it would be simple, I can look into at least fixing that.

To me this seems like a bug in the upstream datafusion implementation. Would it not be better to address the bug there? e.g make that implementation have correct behavior around dictionary encoded types.

I agree this is mostly a DataFusion bug, but it's at least partially a comet thing for choosing to make the dictionaries in the first place. Mostly I guess I'm asking if it's worth coming up with a workaround for this specific issue or more generally any dictionary related issue that will let things work until DataFusion handles things correctly, which might be non-trivial to fix.

Seems like that would cause an unnecessary copy in the case of the make_array. To me it seems like we should just be "unpacking the dictionary as part of the data getting writing into the new array data structure. And other expressions might be able to do other optimizations by having data in the dictionary format for example #504

I agree that is the best case scenario. Since doing the unpacking as part of creating the array data structure might be a non-trivial fix (at least for someone like me who's just learning about DataFusion), is it worth making a way to opt-in to a workaround that will pre-unravel a dictionary before sending it into a DataFusion function that Comet expressions can opt-in to until a better more permanent fix is figured out.

Ok I learned a little more about DataFusion so I think I understand what the options are now. ScalarUDFs support type coercion, which will automatically cast each expression to the right type, and that gets inserted in analysis. Because of this, there's nothing technically "wrong" or buggy in DataFusion. Possibly just sub-optimal of doing a cast instead of smartly handling mixed types. Obviously Comet isn't using the DataFusion analyzer, so that will never happen automatically here. And the type coercion theoretically handles the differences between dictionaries/non-dictionaries.
Based on this, it seems like there's two options:

Add some form of support for handling ScalarUDF type coercion automatically
Don't use ScalarUDFs when type coercion may be needed and only use custom implemented expressions


Implement a desugaring transformation

CertLogic has as few operations as possible to mimise the amount of work needed to implement these across target platforms (/programming languages), and testing them.
As a nod to the developer experience (DX), we can implement a desugaring transformation that desugars operation such or or, and !== (which are not part in CertLogic, but would be easy to specify in terms of CertLogic operations) to CertLogic. E.g. l or r === !(!l and !r), and !== === !===.

Done per PR #74


Fixed thread scheduling bug in CLR Thread Pool

The CLR thread pool now tracks the number of requested threads instead of number of work items in the queue when deciding whether or not to try adding an new worker.

@jkoritzinsky,
Thanks for having already signed the Contribution License Agreement. Your agreement was validated by .NET Foundation. We will now review your pull request.
Thanks,
.NET Foundation Pull Request Bot


Self-update fails if start folder is repository

If a shell defaults to a git repository folder instead of the home folder, and the git repository has unstaged changes, then self-update fails.
[Oh My Zsh] Would you like to update? [Y/n]:
Updating Oh My Zsh
error: cannot pull with rebase: You have unstaged changes.
error: please commit or stash them.
There was an error updating. Try again later?

I suggest storing current working directory, then moving to a safe location for doing the update, then moving back to user's preferred "start folder" after update.

I'm not sure I follow. The upgrade process always cds into the ohmyzsh folder to do the pull rebase. The error you're seeing is that you have unstaged changes, but on the ohmyzsh folder.

I can confirm that this is not the case. There are no unstaged changes in ~/.oh-my-zsh. However, I do cd around in my .zshrc. Maybe that's what causes the problem.

As I said, the upgrade script cds into the ohmyzsh folder right before the pull rebase: https://github.com/robbyrussell/oh-my-zsh/blob/05b617066ba5a37ef0c533385efd6e232a387b8f/tools/upgrade.sh#L24-L25
I don't know how cds in your zshrc file could affect the process... Can you tell me what ohmyzsh commit you have checked out?

Hmm. Hope this isn't just some error I've caused myself. I'm at commit 16bfd6f(git log below).

What does cd $ZSH; git status -s show?

cd $ZSH; git status -s:


Data disk attach

Update compute data-disk command

add data-disk attach subcommand attach an existing data disk to the selected virtual machine

LGTM

@schaefi - sorry to be the spoilsport again, but...
you're using ServiceManagementService#add_data_disk in the attach function; can you verify this works with both of these scenarios:

a 'new' data disk, e.g. a page blob uploaded with storage disk upload,
a 'detached' disk

I just want to make sure the metadata gets handled properly in both cases, and that we don't need to use ServiceManagementService#update_data_disk on the detached disk, which already has data-disk metadata.

I have tested the first scenario in my original pull request where I had changed the create call to upload a vhd and register that as disk. The disk could be attached by the code submitted here
I'll test re-attaching a detached disk now

{
  "is_attached": false,
  "name": "suse-products-suma-0-201608181134280818"
}

azurectl --account publish compute data-disk show --disk-name suse-products-suma-0-201608181134280818

{
  "data-disk": {
    "affinity_group": "",
    "attached_to": {},
    "has_operating_system": "",
    "is_corrupted": "",
    "label": "",
    "location": "West US",
    "logical_disk_size_in_gb": "8 GB",
    "media_link": "https://susesuselinuxenterprises.blob.core.windows.net/vm-images/suma-data-disk-2016-08-18T11_34_28.223418.vhd",
    "name": "suse-products-suma-0-201608181134280818",
    "os": "",
    "source_image_name": ""
  }
}

azurectl --account publish compute data-disk attach --cloud-service-name suse-products --instance-name suma --disk-name suse-products-suma-0-201608181134280818

azurectl --account publish compute data-disk list

{
  "is_attached": true,
  "name": "suse-products-suma-0-201608181134280818"
}

azurectl --account publish compute data-disk show --disk-name suse-products-suma-0-201608181134280818

{
  "data-disk": {
    "affinity_group": "",
    "attached_to": {
      "deployment_name": "suse-products",
      "hosted_service_name": "suse-products",
      "role_name": "suma"
    },
    "has_operating_system": "",
    "is_corrupted": "",
    "label": "",
    "location": "West US",
    "logical_disk_size_in_gb": "8 GB",
    "media_link": "https://susesuselinuxenterprises.blob.core.windows.net/vm-images/suma-data-disk-2016-08-18T11_34_28.223418.vhd",
    "name": "suse-products-suma-0-201608181134280818",
    "os": "",
    "source_image_name": ""
  }
}

For me this all works nicely.
Again I have tested my first pull request extensively in real scenarios
no trust in schaefi coding :smile:


Mobile. 'Про нас' page view

As a user I want to be able to see the site "Про нас" page in the mobile view so that I could see it from my phone
Mockups:
https://www.figma.com/file/d2105v4IAq59MjeB01DfsR/Навчай-українською-|-Delivery?node-id=1805%3A189353
Acceptance criteria

User sees the current state of web page in the mobile view
User can scroll down the page, vertical scroll is shown when page is scrolled down-scrolled up

BA notes:

Texts should be stretched across the width of the mobile screen
Design bug

Photos should be stretched across the width of the mobile screen like below:

@marianben kindly take those notes into consideration

The same bug #603


[Bug]: Correct unsuscribe emails (dev and user mailing lists)

What happened?
Page : https://beam.apache.org/community/contact-us/
Paragraph Title : Subscribe and Unsubscribe:
The mail used in unsuscribe links is<EMAIL_ADDRESS>The correct mail is<EMAIL_ADDRESS>and<EMAIL_ADDRESS>Issue Priority
Priority: 3 (minor)
Issue Components

[ ] Component: Python SDK
[ ] Component: Java SDK
[ ] Component: Go SDK
[ ] Component: Typescript SDK
[ ] Component: IO connector
[ ] Component: Beam YAML
[ ] Component: Beam examples
[ ] Component: Beam playground
[ ] Component: Beam katas
[X] Component: Website
[ ] Component: Spark Runner
[ ] Component: Flink Runner
[ ] Component: Samza Runner
[ ] Component: Twister2 Runner
[ ] Component: Hazelcast Jet Runner
[ ] Component: Google Cloud Dataflow Runner

.take-issue


Input lag/latency when using combo keys

Not sure if anyone else has experienced this issue, but I seem to have lag or the modifier key doesn't register. It seems to happen when I'm typing faster and I didn't experience this when using Seil in mac 10.11.
Example: Press cmd+a, end up typing a
MacOS 10.12
Also using a poker 3 keyboard, but not sure if that affects anything.
karabiner.json
{
  "profiles": [
    {
      "name": "Default profile",
      "selected": true,
      "simple_modifications": {
        "caps_lock": "escape",
        "escape": "grave_accent_and_tilde",
        "left_command": "left_option",
        "left_option": "left_command"
      }
    }
  ]
}

Hey, I am having many issues with my poker 3 too, have got the fn key to work?


Cannot destructure property connId of 'undefined' or 'null'.

Describe the bug
Trying to save a PostgreSQL to JSON or CSV
To Reproduce
Steps to reproduce the behavior:

Run any Query
Try to save as JSON/CSV
Cannot destructure property connId of 'undefined' or 'null'.

Expected behavior
Expected to save....
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):

SQLTools Version: v0.19
VSCode Version: 1.34.0
Dialect Used: SQLite
OS: Linux Ubuntu

Additional context
Any queries I try to run as well seem to bug out and fail. I've tried some of the fixes in here including checking for "." somewhere in the name...

Close in favor of #259.
It's the same issue.


[Docs] Improving Meshery Extensibility Providers Docs

Current State
Currently, when reading through Meshery Providers Docs, people tend to get lost in the middle of the documentation. At some point consider decoupling the documentation for example Guides and Tutorial and Extensions. The docs of explanation should explain for example what we do and why we do this in this way.
Desired State
Identify which issues are on the page and provide better alternatives to make a boundary for example around Guides and Tutorial, Integration, and Extensions.
Reference
Meshery Docs Persona
Writing a good explanation docs

Contributor Guides and Handbook

📚 Instructions for contributing to documentation

Meshery documentation site and source

🛠 Meshery Build & Release Strategy
🎨 Wireframes and designs for Meshery UI in Figma (open invite)
🙋🏾🙋🏼 Questions: Discussion Forum and Community Slack

@jerensl Can you assign me this?

@jerensl can i work on this ?

Hi @jerensl ,I already have Meshery built locally. I’ll Understand Meshery Providers.Split docs into Guides, Integrations, and Extensions.Add clear explanations for each section.


calico-node 3.0.2 enters CrashLoopBackOff

Installing calico-node v3.0.2 with the guide on https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/hosted
Expected Behavior
calico pods are in Running state.
Current Behavior
$ kubectl get pod -n kube-system
NAME                                       READY     STATUS             RESTARTS   AGE
calico-kube-controllers-5f748b7f66-t8665   1/1       Running            0          33m
calico-node-9lmjx                          1/2       CrashLoopBackOff   11         33m
calico-node-snv5h                          1/2       CrashLoopBackOff   11         33m
calico-node-t8w5z                          1/2       CrashLoopBackOff   11         33m

corresponding kubelet log
Feb 25 15:27:46 k8s401 kubelet[4580]: I0225 15:27:46.238452    4580 kuberuntime_manager.go:758] checking backoff for container "install-cni" in pod "calico-node-snv5h_kube-system(12be2642-19f8-11e8-ba54-525400cecc16)"
Feb 25 15:27:46 k8s401 kubelet[4580]: I0225 15:27:46.238632    4580 kuberuntime_manager.go:768] Back-off 5m0s restarting failed container=install-cni pod=calico-node-snv5h_kube-system(12be2642-19f8-11e8-ba54-525400cecc16)
Feb 25 15:27:46 k8s401 kubelet[4580]: E0225 15:27:46.238671    4580 pod_workers.go:186] Error syncing pod 12be2642-19f8-11e8-ba54-525400cecc16 ("calico-node-snv5h_kube-system(12be2642-19f8-11e8-ba54-525400cecc16)"), skipping: failed to "StartContainer" for "install-cni" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=install-cni pod=calico-node-snv5h_kube-system(12be2642-19f8-11e8-ba54-525400cecc16)"

Possible Solution

it seemd that the container install-cni causing the problem
Steps to Reproduce (for bugs)

Context

Your Environment

Calico version v3.0.2
Orchestrator version (e.g. kubernetes, mesos, rkt): k8s v1.9.3
Operating System and version: ubuntu 16.04
Link to your project (optional):

here is the detailed log:
calico-node.log
and
install-cni.log

Thanks for opening this separate issue @gjmzj
I see from the kubelet log why you're saying it is the install-cni container.  I'm thinking maybe that log message is misleading and it might be another problem.  Before we dig into that though I want to get some clarification.  The PR you originally commented on was around doing the v2.6.x to v3.0 migration, is that still the case here?  This issue you submitted seems to indicate a fresh install.  If it is a "fresh" install I'm wondering, have you reused your etcd datastore?  I'm asking that because of the calico-node logs repeatedly having
2017-12-19 20:41:43.572 [INFO][100] felix.go 666: Waiting for the datastore to be ready
2017-12-19 20:41:43.573 [WARNING][100] felix.go 672: Global datastore 'Ready' flag set to false, waiting...

If you've reused your etcd datastore then I think there is previous Calico etcd data that needs to be removed before deploying Calico again.
If the above doesn't help then could you please do a kubectl describe -n kube-system pod <calico node pod> and add its output, I'm looking for anything that indicates why k8s thinks there is a problem with install-cni.

@tmjd Thanks for your time
my cluster is a fresh install, so is the etcd datastore.
i've just re-installed again, here is the log:
$ kubectl describe pod -n kube-system calico-node-7dfnq
Name:         calico-node-7dfnq
Namespace:    kube-system
Node:         <IP_ADDRESS>/<IP_ADDRESS>
Start Time:   Tue, 27 Feb 2018 08:43:01 +0800
Labels:       controller-revision-hash=384064918
              k8s-app=calico-node
              pod-template-generation=1
Annotations:  scheduler.alpha.kubernetes.io/critical-pod=
              scheduler.alpha.kubernetes.io/tolerations=[{"key": "dedicated", "value": "master", "effect": "NoSchedule" },
 {"key":"CriticalAddonsOnly", "operator":"Exists"}]

Status:         Running
IP:             <IP_ADDRESS>
Controlled By:  DaemonSet/calico-node
Containers:
  calico-node:
    Container ID:   docker://2920b4dd387d66a00e73da0b31471858a7f78402c2a383ca3920983869c7ace5
    Image:          calico/node:v3.0.2
    Image ID:       docker-pullable://calico/node@sha256:9df9de1503f6312f42ccb0f13e9e226f11a351bd5ad4a2ef13cde0a4d7b497d8
    Port:           <none>
    State:          Running
      Started:      Tue, 27 Feb 2018 08:43:19 +0800
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   http-get http://:9099/liveness delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  http-get http://:9099/readiness delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      ETCD_ENDPOINTS:                     <set to the key 'etcd_endpoints' of config map 'calico-config'>  Optional: false
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      CALICO_DISABLE_FILE_LOGGING:        true
      CALICO_K8S_NODE_REF:                 (v1:spec.nodeName)
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      CALICO_IPV4POOL_CIDR:               <IP_ADDRESS>/16
      CALICO_IPV4POOL_IPIP:               off
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_IPINIPMTU:                    1440
      ETCD_CA_CERT_FILE:                  <set to the key 'etcd_ca' of config map 'calico-config'>    Optional: false
      ETCD_KEY_FILE:                      <set to the key 'etcd_key' of config map 'calico-config'>   Optional: false
      ETCD_CERT_FILE:                     <set to the key 'etcd_cert' of config map 'calico-config'>  Optional: false
      IP:                                 autodetect
      IP_AUTODETECTION_METHOD:            interface=ens3
      FELIX_HEALTHENABLED:                true
    Mounts:
      /calico-secrets from etcd-certs (rw)
      /lib/modules from lib-modules (ro)
      /var/run/calico from var-run-calico (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-5wp6l (ro)
  install-cni:
    Container ID:  docker://1cf11319483fc9a6e5d5b1fcd7a996b93d98cd4870226a502af50ac7b8c56723
    Image:         calico/cni:v2.0.1
    Image ID:      docker-pullable://calico/cni@sha256:a53b174afd4b0b4b1a9791dcc363800af1e31e59ee487b1afe8bf10113d87d1e
    Port:          <none>
    Command:
      /install-cni.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 27 Feb 2018 08:44:12 +0800
      Finished:     Tue, 27 Feb 2018 08:44:12 +0800
    Ready:          False
    Restart Count:  3
    Environment:
      CNI_CONF_NAME:       10-calico.conflist
      ETCD_ENDPOINTS:      <set to the key 'etcd_endpoints' of config map 'calico-config'>      Optional: false
      CNI_NETWORK_CONFIG:  <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
    Mounts:
      /calico-secrets from etcd-certs (rw)
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-5wp6l (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /root/local/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  etcd-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/calico/ssl
    HostPathType:
  calico-node-token-5wp6l:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-5wp6l
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
Events:
  Type     Reason                 Age               From                   Message
  ----     ------                 ----              ----                   -------
  Normal   SuccessfulMountVolume  1m                kubelet, <IP_ADDRESS>  MountVolume.SetUp succeeded for volume "var-run-calico"
  Normal   SuccessfulMountVolume  1m                kubelet, <IP_ADDRESS>  MountVolume.SetUp succeeded for volume "cni-bin-dir"
  Normal   SuccessfulMountVolume  1m                kubelet, <IP_ADDRESS>  MountVolume.SetUp succeeded for volume "cni-net-dir"
  Normal   SuccessfulMountVolume  1m                kubelet, <IP_ADDRESS>  MountVolume.SetUp succeeded for volume "lib-modules"
  Normal   SuccessfulMountVolume  1m                kubelet, <IP_ADDRESS>  MountVolume.SetUp succeeded for volume "etcd-certs"
  Normal   SuccessfulMountVolume  1m                kubelet, <IP_ADDRESS>  MountVolume.SetUp succeeded for volume "calico-node-token-5wp6l"
  Normal   Pulling                1m                kubelet, <IP_ADDRESS>  pulling image "calico/node:v3.0.2"
  Normal   Pulled                 1m                kubelet, <IP_ADDRESS>  Successfully pulled image "calico/node:v3.0.2"
  Normal   Created                1m                kubelet, <IP_ADDRESS>  Created container
  Normal   Started                1m                kubelet, <IP_ADDRESS>  Started container
  Normal   Pulling                1m                kubelet, <IP_ADDRESS>  pulling image "calico/cni:v2.0.1"
  Normal   Pulled                 1m                kubelet, <IP_ADDRESS>  Successfully pulled image "calico/cni:v2.0.1"
  Normal   Created                1m (x3 over 1m)   kubelet, <IP_ADDRESS>  Created container
  Normal   Started                1m (x3 over 1m)   kubelet, <IP_ADDRESS>  Started container
  Warning  BackOff                47s (x4 over 1m)  kubelet, <IP_ADDRESS>  Back-off restarting failed container
  Normal   Pulled                 32s (x3 over 1m)  kubelet, <IP_ADDRESS>  Container image "calico/cni:v2.0.1" already present on machine

@tmjd some more info:
it seems that everything is working well if i just ignore this problem.
$ kubectl get pod --all-namespaces -o wide
NAMESPACE     NAME                                       READY     STATUS             RESTARTS   AGE       IP               NODE
default       busy-5cff6d878d-7826v                      1/1       Running            1          1h        <IP_ADDRESS>    <IP_ADDRESS>
default       busy-5cff6d878d-9qbkk                      1/1       Running            1          1h        <IP_ADDRESS>   <IP_ADDRESS>
default       busy-5cff6d878d-fkhv4                      1/1       Running            1          1h        <IP_ADDRESS>     <IP_ADDRESS>
default       nginx-7587c6fdb6-76vhw                     1/1       Running            0          1h        <IP_ADDRESS>    <IP_ADDRESS>
default       nginx-7587c6fdb6-jvthb                     1/1       Running            0          1h        <IP_ADDRESS>     <IP_ADDRESS>
default       nginx-7587c6fdb6-s2pfp                     1/1       Running            0          1h        <IP_ADDRESS>     <IP_ADDRESS>
kube-system   calico-kube-controllers-5f748b7f66-zz67s   1/1       Running            0          2h        <IP_ADDRESS>     <IP_ADDRESS>
kube-system   calico-node-f87zw                          1/2       CrashLoopBackOff   15         56m       <IP_ADDRESS>     <IP_ADDRESS>
kube-system   calico-node-rrqfg                          1/2       CrashLoopBackOff   34         2h        <IP_ADDRESS>     <IP_ADDRESS>
kube-system   calico-node-zsgxv                          1/2       CrashLoopBackOff   34         2h        <IP_ADDRESS>     <IP_ADDRESS>
kube-system   kube-dns-566c7c77d8-5xxln                  3/3       Running            0          1h        <IP_ADDRESS>     <IP_ADDRESS>

and i tested the network policy, it worked too

It is not too surprising that things are 'working' even with this failure, since it looks like the CNI config and binaries are being installed and calico-node is running fine.
Could you try exec'ing into one of the calico-node pods and then run ls /calico-secrets and stat -c%y /calico-secrets/etcd-cert and check the return code on both?  Those are a couple of the commands ran in install-cni after the Done configuring CNI... and I'm wondering if somehow they are failing.
Another thing that would be interesting to try is to remove the file /etc/cni/net.d/10-calico.conflist from one of your nodes and see if it is replaced.  If it is not replaced (after a minute) then delete the calico-node pod that is running on that host so that it is restarted on that host, you should see the file replaced.

thanks for your tips @tmjd
after reading the script install-cni.sh, i found the problem, and solved.
in calico.yaml,   i mistakenly changed the names of TLS secrets files.
changing it to default value, it worked.
  # If you're using TLS enabled etcd uncomment the following.
  # You must also populate the Secret below with these files.
  etcd_ca: ""   # "/calico-secrets/etcd-ca"
  etcd_cert: "" # "/calico-secrets/etcd-cert"
  etcd_key: ""  # "/calico-secrets/etcd-key"

Thanks for reporting back. I'm going to close this issue since you figured it out.

hello my error for pods calico
root@kubemaster:/# kubectl get pod -n kube-system
NAME                                       READY   STATUS              RESTARTS   AGE
calico-kube-controllers-7c4fb94b96-99gjr   0/1     CrashLoopBackOff    24         39m
calico-node-b7dpb                          0/1     CrashLoopBackOff    23         39m
coredns-6955765f44-qdx99                   0/1     ContainerCreating   0          54m
coredns-6955765f44-vg9hm                   0/1     ContainerCreating   0          54m
etcd-kubemaster                            1/1     Running             4          54m
kube-apiserver-kubemaster                  1/1     Running             4          54m
kube-controller-manager-kubemaster         1/1     Running             4          54m
kube-proxy-rzxtg                           1/1     Running             4          54m
kube-scheduler-kubemaster                  1/1     Running             4          54m

Events:
  Type     Reason            Age                 From                 Message
  ----     ------            ----                ----                 -------
  Warning  FailedScheduling  46m (x5 over 46m)   default-scheduler    0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled         46m                 default-scheduler    Successfully assigned kube-system/calico-kube-controllers-7c4fb94b96-99gjr to kubemaster
  Normal   Pulling           45m                 kubelet, kubemaster  Pulling image "calico/kube-controllers:v3.10.2"
  Normal   Pulled            45m                 kubelet, kubemaster  Successfully pulled image "calico/kube-controllers:v3.10.2"
  Normal   Started           44m (x4 over 45m)   kubelet, kubemaster  Started container calico-kube-controllers
  Warning  Unhealthy         44m (x4 over 45m)   kubelet, kubemaster  Readiness probe failed: Failed to read status file status.json: open status.json: no such file or directory
  Normal   Pulled            43m (x4 over 45m)   kubelet, kubemaster  Container image "calico/kube-controllers:v3.10.2" already present on machine
  Normal   Created           43m (x5 over 45m)   kubelet, kubemaster  Created container calico-kube-controllers
  Warning  BackOff           30m (x64 over 45m)  kubelet, kubemaster  Back-off restarting failed container
  Warning  FailedMount       28m                 kubelet, kubemaster  MountVolume.SetUp failed for volume "calico-kube-controllers-token-k7kb8" : failed to sync secret cache: timed out waiting for the condition
  Normal   SandboxChanged    28m                 kubelet, kubemaster  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled            26m (x4 over 28m)   kubelet, kubemaster  Container image "calico/kube-controllers:v3.10.2" already present on machine
  Normal   Created           26m (x4 over 28m)   kubelet, kubemaster  Created container calico-kube-controllers
  Normal   Started           26m (x4 over 28m)   kubelet, kubemaster  Started container calico-kube-controllers
  Warning  Unhealthy         26m (x4 over 28m)   kubelet, kubemaster  Readiness probe failed: Failed to read status file status.json: open status.json: no such file or directory
  Warning  BackOff           22m (x21 over 28m)  kubelet, kubemaster  Back-off restarting failed container
  Normal   SandboxChanged    20m                 kubelet, kubemaster  Pod sandbox changed, it will be killed and re-created.
  Normal   Started           20m                 kubelet, kubemaster  Started container calico-kube-controllers
  Normal   Created           20m                 kubelet, kubemaster  Created container calico-kube-controllers
  Normal   Pulled            20m                 kubelet, kubemaster  Container image "calico/kube-controllers:v3.10.2" already present on machine
  Normal   SandboxChanged    19m                 kubelet, kubemaster  Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy         18m                 kubelet, kubemaster  Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:345: starting container process caused "process_linux.go:91: executing setns process caused \"exit status 21\"": unknown
  Normal   Pulled            17m (x4 over 19m)   kubelet, kubemaster  Container image "calico/kube-controllers:v3.10.2" already present on machine
  Normal   Created           17m (x4 over 19m)   kubelet, kubemaster  Created container calico-kube-controllers
  Normal   Started           17m (x4 over 19m)   kubelet, kubemaster  Started container calico-kube-controllers
  Warning  Unhealthy         17m (x3 over 19m)   kubelet, kubemaster  Readiness probe failed: Failed to read status file status.json: open status.json: no such file or directory
  Warning  BackOff           17m (x8 over 19m)   kubelet, kubemaster  Back-off restarting failed container
  Normal   SandboxChanged    15m                 kubelet, kubemaster  Pod sandbox changed, it will be killed and re-created.
  Warning  Unhealthy         14m                 kubelet, kubemaster  Readiness probe errored: rpc error: code = Unknown desc = container not running (80262d1e9d6564ece32424e7e539011b837cbe711ef9ef9658b79fc46bbf6c52)
  Normal   Pulled            13m (x4 over 15m)   kubelet, kubemaster  Container image "calico/kube-controllers:v3.10.2" already present on machine
  Normal   Created           13m (x4 over 15m)   kubelet, kubemaster  Created container calico-kube-controllers
  Normal   Started           13m (x4 over 15m)   kubelet, kubemaster  Started container calico-kube-controllers
  Warning  Unhealthy         13m (x2 over 14m)   kubelet, kubemaster  Readiness probe failed: Failed to read status file status.json: open status.json: no such file or directory
  Warning  BackOff           37s (x66 over 15m)  kubelet, kubemaster  Back-off restarting failed container

using command for run yml
kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml

Hi @rezabojnordi, could you please open a new issue?
I would also add: v3.0 is a very old version of Calico; many improvements and bug fixes have been added to Calico since then. I would suggest trying again with the latest version of Calico (v3.11.1) if that's possible for you.