Stackstac 0.5.0 produces bad pixel values for Sentinel-2

When querying Sentinel-2 images from AWS (Element-84), using stackstac 0.5.0 pixel values are significantly different than with 0.4.3. I believe 0.4.3 is giving the correct values. Inspection of some values makes it look like 0.5.0 is subtracting 1000 (maybe incorrectly applying a PB4 correction?) then dividing by 10,000.
Minimal reproducible example:
import stackstac
from pystac_client import Client
geom = {'type': 'Polygon',
 'coordinates': [[(-126.37, 50.43),
   (-126.35, 50.43),
   (-126.35, 50.45),
   (-126.37, 50.45),
   (-126.37, 50.43)]]}
start_date = '2020-07-01'
end_date = '2020-07-03'
url = "https://earth-search.aws.element84.com/v1"
cat = Client.open(url)
date_query = start_date+'/'+end_date
params = {
"intersects": geom,
"collections": ["sentinel-2-l2a"],
"datetime": date_query,}
search = cat.search(**params)
items = search.item_collection()
stack = stackstac.stack(
    items,
    assets=['blue'],
    bounds_latlon=[-126.37,50.43,-126.35,50.45],
    resolution=10,
    epsg=32609)
print(stack[0,0,0,0].to_numpy())

This code with stackstac 0.4.3 give 3176.0 (a reasonable DN for Sentinel-2).
With stackstac 0.5.0, it gives 0.2176.

It should be due to the fact that the newest version applies automatically offset and scale to the data, so that you get the reflectance value and not the digital number. However, in this case there seems to be an issue, since the offset has been introduced with data starting from 2023 onwards.
Related issues to the new processing baseline 4.0 which introduced the offset:
https://github.com/Element84/earth-search/issues/9
https://github.com/Element84/earth-search/issues/23

Ah ok, I see! I did not see the addition of the default rescale=True to stack(). We had our own rescaling in place, so this was a breaking change.
Looking at the first item in the query above, it has s2:processing_baseline = 05.00 so the offset should be applied. The property earthsearch:boa_offset_applied = True. However, raster:bands offset=-0.1. So if stackstac is using the raster:bands property, then it looks like it is wrong for this item. You could use earthsearch:boa_offset_applied but I suppose that is unique the Element-84 STAC.
In our own processing, we have been using the earthsearch:boa_offset_applied. It does complicate things as you have to check both the processing baseline, and whether earthsearch:boa_offset_applied exists and is False for a given item. The property doesn't exist for some items, in which case it becomes None instead of False after calling stack().

Setting rescale=False gives you the correct DN value. So, as you understood, the issue is with the Element84 metadata.
import stackstac
from pystac_client import Client
geom = {'type': 'Polygon',
 'coordinates': [[(-126.37, 50.43),
   (-126.35, 50.43),
   (-126.35, 50.45),
   (-126.37, 50.45),
   (-126.37, 50.43)]]}
start_date = '2020-07-01'
end_date = '2020-07-03'
url = "https://earth-search.aws.element84.com/v1"
cat = Client.open(url)
date_query = start_date+'/'+end_date
params = {
"intersects": geom,
"collections": ["sentinel-2-l2a"],
"datetime": date_query,}
search = cat.search(**params)
items = search.item_collection()
stack = stackstac.stack(
    items,
    assets=['blue'],
    bounds_latlon=[-126.37,50.43,-126.35,50.45],
    resolution=10,
    epsg=32609,
    rescale=False
    )
print(stack[0,0,0,0].to_numpy())
> 3176.0

FYI Element 84 is aware about these confusions and announced aligning processing baseline and metadata in the coming months :)

Great, thanks for the replies! I'll close the issue

FYI this has not been fixed and is quite confusing on the stackstac side since there is no warning or notification that rescaling is happening by default. Given that it has been the better part of a year without resolution on the metadata side, would it be worth making some sort of change on the stackstac side? Could be:
-a note somewhere in the documentation
-a warning if rescale=True and the asset is Sentinel-2 (not sure how hard this would be to implement...)
-changing the default for rescale to be False (this would obviously require a warning/notification to users)


Security concern with HMR?

Describe the bug
Normally when you try to import and use $env/dynamic/private in a public context an error will be thrown.
Cannot import $env/dynamic/private into public-facing code:
However there is a way you can temporarily get around this behavior by starting the app and refreshing via HMR.
Reproduction

Clone https://github.com/Jojoshua/sveltekit-double-mount/tree/PrivateENV
Start with the dev script
Uncomment lines 4 and 5 from src/routes/+page.svelte and save
Observe the private_env is printed in the console

Logs
No response
System Info
System:
    OS: Windows 10 10.0.19043
    CPU: (16) x64 AMD Ryzen 7 1700X Eight-Core Processor
    Memory: 4.79 GB / 15.93 GB
  Binaries:
    Node: 18.9.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.18 - ~\AppData\Roaming\npm\yarn.CMD
    npm: 8.4.1 - ~\AppData\Roaming\npm\npm.CMD
  Browsers:
    Chrome: 106.0.5249.103
    Edge: Spartan (44.19041.1266.0), Chromium (106.0.1370.37)
    Internet Explorer: 11.0.19041.1566
  npmPackages:
    @sveltejs/adapter-auto: 1.0.0-next.80 => 1.0.0-next.80
    @sveltejs/kit: 1.0.0-next.504 => 1.0.0-next.504
    @sveltejs/package: 1.0.0-next.5 => 1.0.0-next.5
    svelte: ^3.50.1 => 3.50.1
    vite: ^3.1.3 => 3.1.3

Severity
annoyance
Additional Information
No response

I wouldn't call this a security concern since this will be caught when building for production. Also I'm not sure how much we can do here given the HMR trickeries, but I agree it's raising an eyebrow.

Yes I'm not sure what the implications are so wanted to bring this to the team to review.
I'm not much concerned with local development but curious about online development environments like stack blitz, etc.

This isn't a security concern, as it only happens in dev. I don't believe there's any way for us to get around it, but I'm totally open to suggestions if someone can think of a way. I'm not sure what implications you're thinking of in online dev environments like StackBlitz... you'd have to have the environment variables exposed to include them in a StackBlitz repo anyway...?

3rding  the "not a security concern". Additional arguments

You really should not use production secrets on your local machine, that's what .env.local is for
The dev-server doesn't listen on external interfaces by default
The dev-server must not be used in production

You'd have to take a series of steps in the wrong direction before this could theoretically be a problem
As for warning the user during dev as opposed to build, this would need to be happen in either handleHotUpdate or transform hook by throwing an error. Right now the check happens during writeBundle. https://github.com/sveltejs/kit/blob/master/packages/kit/src/exports/vite/index.js#L358

Good info @dominikg . In my case I was testing how I could make configuration replacements (for a library) at compile time by using import { env as public_env } from '$env/dynamic/public' but ended up testing dynamic/private also.
I discovered the replacements don't actually work when packaging (similar to https://github.com/sveltejs/kit/issues/3480) so I moved away from env entirely.
I don' t currently have an exploit to take advantage of this behavior but just had some general follow-on thoughts...

Developers are increasingly the targets of cyber-attacks & the source of breaches
Once a developer finds a way of "making it work" they will stick with it
An attacker will often use minor "quirks" as an advantage to escalate and create a more sophisticated attack than otherwise possible

I'm happy for a member to close this issue whenever it makes sense. Thanks for the quick review guys!

Developers are increasingly the targets of cyber-attacks & the source of breaches
Once a developer finds a way of "making it work" they will stick with it
An attacker will often use minor "quirks" as an advantage to escalate and create a more sophisticated attack than otherwise possible

I don't see any way this could be used to "expose" a private environment variable. If someone has the level of access required to

Visit a specific route (on the dev server)
Trigger HMR
Parse the HTML for a specific value

They could just... read the environment variable from the environment. Or screw you over in any number of other ways.
Ideally, this wouldn't happen, but I really don't know a way around it, and I don't think it's ever going to cause an issue.

Closing as this isn't a security concern - as pointed out, dev shouldn't be used in prod environments, and if someone has access to your dev environment, worse things will happen than having access to the secret in the running app.


Support generating non-AWS client

Description
These changes are to support generating a non-AWS typescript client. There are changes in the code generation logic to skip pieces that apply only to AWS services. Some typescript dependencies also needed updates.
This is initial prototype work for smithy-typescript-ssdk-demo.
Testing
Using these changes I successfully generated a working non-AWS typescript client for the bootleg-service in smithy-typescript-ssdk-demo.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

One thing I wanted to improve - EndpointsInputConfig.endpoint is optional. So when a customer of the BootlegClient in instantiating an instance of the client and doesn't provide an endpoint in the constructor, it is not a compile error. Since it is marked optional in Endpoints.ts and that's used for AWS service clients where it should be optional, I couldn't change this easily. I'd like to discuss ideas on how to do this.

One thing I wanted to improve - EndpointsInputConfig.endpoint is optional. So when a customer of the BootlegClient in instantiating an instance of the client and doesn't provide an endpoint in the constructor, it is not a compile error. Since it is marked optional in Endpoints.ts and that's used for AWS service clients where it should be optional, I couldn't change this easily. I'd like to discuss ideas on how to do this.

You can do something like this:
constructor(c: EndpointsInputConfig & { endpoint : string }) {
 ...
}

This will make endpoint required for your constructor, even though it's optional in EndpointsInputConfig.

Moved to upstream repo - https://github.com/aws/aws-sdk-js-v3/pull/2222


Out of map range behaviour ?

Hi, and thanks for the amazing work on this module !
I'm currently debugging a nasty problem in my project, and my last idea before going to bed is that your SpatialHash might have strange behaviour if a query range goes out of map dimensions.
Whether it's the case or not, maybe this should be specified in Readme (or maybe I'm the only one wondering this).
Thank you !

It'll throw an exception as soon as a non-existing bucket gets called, i.e. it must be inside the hash's boundaries.
You can mitigate this by increasing the hash's width & height by an approximation of the object's width & height respectively.
I'll specify this in the readme now, thanks for reporting 👍

thanks for your quick answer.
Now that you mention increasing hash's dimensions, am I correct assuming
negative values are supported ? Thanks again
2017-02-27 0:08 GMT+01:00 Luka<EMAIL_ADDRESS>
It'll throw an exception as soon as a non-existing bucket gets called,
i.e. it must be inside the hash's boundaries.
You can mitigate this by increasing the hash's width & height by an
approximation of the object's width & height respectively.
I'll specify this in the readme now, thanks for reporting 👍
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Luka967/SpatialHash/issues/1#issuecomment-282596407,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtoPLiaA3lNQjGRz0XSR5H2sLPWZhKbks5rggXXgaJpZM4MMi9W
.

Negative values are supported, don't worry.

Thanks again
Le 27 févr. 2017 11:27 AM, "Luka"<EMAIL_ADDRESS>a écrit :

Negative values are supported, don't worry.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Luka967/SpatialHash/issues/1#issuecomment-282683000,
or mute the thread
https://github.com/notifications/unsubscribe-auth/APtoPLv0Zdmsx6FtUUlr44E78HV88phWks5rgqUdgaJpZM4MMi9W
.

Excuse me for being stupid, I'm sure I'm overlooking something easy, but I don't understand this behaviour
 'use strict';

const SpatialHash = require('spatialhash-2d');
var occupants = new SpatialHash({x:0,y:0,w:51,h:54}, 6);

var testItem = {range:{x: 3, y : 3, w:0,h:0}};
occupants.insert(testItem);

var w = occupants.query({x:5,y:6,w:20,h:12});
for (let i = 0; i <w.length; i++){
    console.log (w[i]);
}

testItem being at x=3 y=3, it shouldn't be found in my query. yet the output is
{ range: { x: 3, y: 3, w: 0, h: 0 },   __b: { hStart: 0, hEnd: 0, vStart: 0, vEnd: 0, id: -9000000000000000 } }
Sorry for bothering you...

 I just read your readme again, and found that x and y are center and w and h are half-length...
Sorry !


'attention.bias' and 'attention.masked_bias' not in hf_layer.state_dict() when converting gpt-neox model to huggingface

Describe the bug
A clear and concise description of what the bug is.
I encounter the following error when I am converting GPTNeoX models to Huggingface using the tools/convert_module_to_hf.py script.
(gpt-neox) johnny@ink-lucy:~/gpt-neox$ bash haveibeentrainedon/wikitext/pilot/convert_to_hf.sh
[2023-08-18 23:37:21,695] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
> building GPT2BPETokenizer tokenizer ...
 > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
Saving weights in fp16 precision...
  0%|                                                                                | 0/24 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "./tools/convert_module_to_hf.py", line 307, in <module>
    hf_model = convert(args.input_dir, loaded_config, args.output_dir)
  File "./tools/convert_module_to_hf.py", line 230, in convert
    state_dict["attention.bias"] = hf_layer.state_dict()["attention.bias"]
KeyError: 'attention.bias'

Expected behavior
Successful conversion.
Proposed solution
If you comment out lines 230 and 231, the script will run through. From an eyeballing of the results, it doesn't seem like language modelling performance seriously degraded. Could this be some code that was supposed to be taken out?
Additional context
This is for a model trained with the config configs/pythia/410m.yml

I realized that the layernorm weights and biases weren't correctly being loaded in, so the lines that I commented out might have caused the problem, or it might be some other problem in the script. Currently trying to find a solution.

transformers==4.31.0
Before transformers==4.31.0:
ipdb> hf_layer.state_dict().keys()
odict_keys(['input_layernorm.weight', 'input_layernorm.bias', 'post_attention_layernorm.weight', 'post_attention_layernorm.bias', 'attention.rotary_emb.inv_freq', 'attention.query_key_value.weight', 'attention.query_key_value.bias', 'attention.dense.weight', 'attention.dense.bias', 'mlp.dense_h_to_4h.weight', 'mlp.dense_h_to_4h.bias', 'mlp.dense_4h_to_h.weight', 'mlp.dense_4h_to_h.bias'])
After transformers==4.30.2 :
ipdb> hf_layer.state_dict().keys()
odict_keys(['input_layernorm.weight', 'input_layernorm.bias', 'post_attention_layernorm.weight', 'post_attention_layernorm.bias', 'attention.bias', 'attention.masked_bias', 'attention.rotary_emb.inv_freq', 'attention.query_key_value.weight', 'attention.query_key_value.bias', 'attention.dense.weight', 'attention.dense.bias', 'mlp.dense_h_to_4h.weight', 'mlp.dense_h_to_4h.bias', 'mlp.dense_4h_to_h.weight', 'mlp.dense_4h_to_h.bias'])
You can find missing 'attention.bias', 'attention.masked_bias'.


Report summary info after applying bulk fixes.

Currently when running dart fix we only report a list of files changed.
Things to consider:

a count of files
a count of fixes applied

/cc @bwilkerson

After  some  conversation, I think we've settled on a simple:
X files fixed in Y seconds.

If we decide that  a fix count (or enumeration of fixed files)  is important we can add them (possibly behind -v).

https://dart-review.googlesource.com/c/sdk/+/166302
Sample output:

(w/ progress while applying fixes.)


Auto refresh when open dashboard

What type of PR is this?

[x] Feature

Description
In this PR we introduce the feature of optionally automatically refresh the dashboard widget query results once we open the dashboard page

We added a feature flag that will allow us to enable the auto-refresh separately for every organisation as you can check it on staging.

This feature flag is not ticked by default (false by default) , but once enabled ... it will refresh the dashboard widgets queries every-time it is accessed.

FYI I noticed that for it to be activated in the beginning you need to reload the page (just for you who is activating it and not another person opening a link...also not you opening a link...just when switching pages just after updating it) so that the frontend gets the new configuration from the backend but this seems okay for me, i guess it is the situation with the other feature flags too.

How is this tested?

[x] Manually

You can check it on staging it is available
Related Tickets & Documents
Closes https://github.com/metr-systems/backlog/issues/3601

@imenattatra After this is reviewed and merged, we can take a look together how to open a PR to Redash :)


Icons for workingsteps >99

In the workingsteps view, the icon for workingsteps with 3+ digits is improperly spaced.

I got 99 problems but a three digit workingstep is one


TimeoutError

websockets connect  TimeError
always  at pycharm/cmd
never at chrome/facebook/homepage/console
alway at chome/other WebSite/console
why?????????
checked for 1000 times,F**K
by the way i use VPN
help me!!!!

Please turn off VPN and try again. If there are still errors, please contact me via: Telegram


converted models producing noise

Some recently-converted are producing random noise, like:

This is:

happening with pytorch 2.x, including 2.0 and 2.1

even with low_cpu_mem_usage monkeypatches on most/all UNetCondition2DModel ctors

only happening when ONNX_WEB_CONVERT_EXTRACT=FALSE

which is necessary for converting some newer models

happening with both fp16 and fp32
happening with SD v1.5 models

does not appear to happen with SDXL

Running a diff between a good and bad copy of the same model, most/all of the weights are different:
INFO:__main__:raw data differs for onnx::Mul_9546: -0.12585449
INFO:__main__:raw data differs for onnx::Add_9547: 0.10827637
INFO:__main__:raw data differs for onnx::MatMul_9548: 0.25146484
INFO:__main__:raw data differs for onnx::MatMul_9549: 0.34448242
INFO:__main__:raw data differs for onnx::MatMul_9550: 0.16455078
INFO:__main__:raw data differs for onnx::MatMul_9557: 0.19995117
INFO:__main__:raw data differs for onnx::MatMul_9558: 0.15942383
INFO:__main__:raw data differs for onnx::MatMul_9559: 0.21325684
INFO:__main__:raw data differs for onnx::MatMul_9560: 0.13708496
INFO:__main__:raw data differs for onnx::MatMul_9567: 0.23217773
INFO:__main__:raw data differs for onnx::MatMul_9568: 0.19250488
INFO:__main__:raw data differs for onnx::MatMul_9569: 0.18237305
INFO:__main__:raw data differs for onnx::Mul_9570: -0.03488159
INFO:__main__:raw data differs for onnx::Add_9571: 2.65625
WARNING:__main__:models have 686 differences

This is true for at least the UNet and VAEs.

When this occurs on Windows, it appears to cause a crash rather than noise:
[2023-12-23 20:16:06,568] ERROR: onnx-web worker: directml MainThread onnx_web.chain.pipeline: error while running stage pipeline, 1 retries left
Traceback (most recent call last):
  File "onnx_web\chain\pipeline.py", line 227, in __call__
  File "onnx_web\chain\source_txt2img.py", line 144, in run
  File "diffusers\pipelines\stable_diffusion\pipeline_onnx_stable_diffusion.py", line 433, in __call__
  File "diffusers\pipelines\stable_diffusion\pipeline_onnx_stable_diffusion.py", line 433, in <listcomp>
  File "onnx_web\diffusers\patches\vae.py", line 79, in __call__
  File "diffusers\pipelines\onnx_utils.py", line 60, in __call__
  File "onnxruntime\capi\onnxruntime_inference_collection.py", line 220, in run
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'/decoder/mid_block/attentions.0/Add_1' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2759)\onnxruntime_pybind11_state.pyd!00007FF863B5DDF2: (caller: 00007FF863B5DB05) Exception(4) tid(5d98) 80070057 The parameter is incorrect.

I believe this is the same issue, and the different error message is due to DirectML.
I've written a new SD converter that uses the same optimum.main_export call that the SDXL converter is using, which seems to work on most models. Currently testing on the models included in the pre-converted set:

Cetus

fails on both v4 and Whalefall

Dreamshaper

works on v8

Elegant Entropy

works on v1.4

Faetastic

fails on v2

Juggernaut

not setup yet, not tested

ReV Animated

works on v1.2.2-EOL

Examples:

Based on the fact that some models fail to convert with both methods, it seems like there might be an issue with the model or somewhere upstream.


High memory usage when doing a RetrieveMultiple for 4 minutes

Hi!
I'm trying to figure out why is there a high memory usage peak that lasts for 4 minutes when doing RetrieveMultiple.
There's a LOH memory usage of 400MB (approx) that disappears after 4 minutes. See image 1 (before) and image 2 (after):

I've done a Snapshot to see what's actually going on with the memory usage and I've found out that there are 385MB of Byte[] that is somehow stucked there for those 4 minutes (see image 3 and image 4).

What the program does is using RetrieveMultiple with PagingInfo to retrieve 30.173 records by pages of size 5.000 records. That's why there are 6 Byte[] of size 60MB (approx). So I'm guessing these LOH objects stored there for 4 minutes are the responses from those queries.
I've checked the code around these lines and I haven't seen any Dispose() on the HttpResponseMessage:
https://github.com/microsoft/PowerPlatform-DataverseServiceClient/blob/dc278e33f6ad4e1b13335d70bb1ee53c6f8d9235/src/GeneralTools/DataverseClient/Client/ConnectionService.cs#L1964
Wouldn't it be necessary in order to tell the GC that the memory allocated for it can be released?
Thanks

Hi, @Ivan-Colomer
What you're seeing is the GC moving though its process,
the only operation in the 1.0.9 and older client that use the WebAPI ( and thus the response payload you call out ) are Create/Update and Delete operations.
All other operations save for a few specific messages go through a WCF protocol currently at a lower level which does have the GC collects invoked.  That code is mostly managed by the underlying Xrm.Sdk.dll bits,  some of which we are moving into the client directly in preparation for a protocol shift.
We will see if we can set this up and confirm this is just the GC doing 'its thing' to verify.
Few specific questions to help us get your repro correct,
What Host OS and .net version are you using? (it matters unfortunately)
And the pattern your suing is around 30 rec's at 5meg-ish a group?
t

Hi @MattB-msft ,
Although I see some GC collects when doing the RetrieveMultiple, these Byte[] arrays doesn't get collected until it passes some minutes (and doing a GC collect manually after that period of time). My main guess was that there is a reference or something to them which doesn't get disposed properly.
I'm very concerned about this topic because I plan to publish an application into an Azure App Service and the ram memory usage is something quite important there.
Currently, the Host OS is a Win 10 Pro x64 and the .Net version is .NET 6.0.8, but I'm planning to run this app on a Linux OS on Azure App Service.
Regarding your last question, the pattern I'm seeing is around 67MB for each RetrieveMultiple of 5.000 records.
Thanks again,

thanks, we will have a look at it and see if we can run down whats going on there.

Hi @Ivan-Colomer , just curious, which tool did you use to track memory usage?
Thanks!

@Ivan-Colomer
The line:
var json = await sResp.Content.ReadAsStringAsync().ConfigureAwait(false);

already reads all the content and buffer it into memory, so there is no need to dispose the HttpResponseMessage.
For large response of HttpClient, it is recommended to use HttpCompletionOption.ResponseHeadersRead to prevent buffer the content, and then you can handle the HttpContent yourself
https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpcompletionoption?view=net-6.0

Hi @Ivan-Colomer , just curious, which tool did you use to track memory usage?
Thanks!

Hi, I used jetBrains dotMemory 2022.2.1

@Ivan-Colomer The line:
var json = await sResp.Content.ReadAsStringAsync().ConfigureAwait(false);

already reads all the content and buffer it into memory, so there is no need to dispose the HttpResponseMessage.
For large response of HttpClient, it is recommended to use HttpCompletionOption.ResponseHeadersRead to prevent buffering the content, and then you can handle the HttpContent by yourself, for example: reuse a pool of small size byte array
https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpcompletionoption?view=net-6.0

You're right, the data would have been buffered automatically and the disposing of the HttpResponseMessage can be skipped, even though it would be nice to always dispose of the HttpResponseMessage once you have finished with using it.
As @MattB-msft said, the RetrieveMultiple operation is sent through a WCF, so the problem must be somewhere else.

@Ivan-Colomer when calling retrieve multiple,
Are you using RetrieveMultipleAsync or just RetrieveMultiple ?

@Ivan-Colomer when calling retrieve multiple, Are you using RetrieveMultipleAsync or just RetrieveMultiple ?

I can confirm the memory issue. It doesn't take minutes, it happens from the first query performed and grows proportionally per query.
The issue occurs when using RetrieveMultipleAsync or ExecuteAsync using a RetrieveMultipleResponse.
I does not occur when using their synchronous counterparts.
I do not know what cause the issue but it leaves one uncollected UTF8BufferedMessageData hanging around per executed query. Not perfect for an Azure environment where you pay for memory consumption :)

@Ivan-Colomer when calling retrieve multiple, Are you using RetrieveMultipleAsync or just RetrieveMultiple ?

I can confirm the memory issue. It doesn't take minutes, it happens from the first query performed and grows proportionally per query. The issue occurs when using RetrieveMultipleAsync or ExecuteAsync using a RetrieveMultipleResponse. I does not occur when using their synchronous counterparts.
I do not know what cause the issue but it leaves one uncollected UTF8BufferedMessageData hanging around per executed query. Not perfect for an Azure environment where you pay for memory consumption :)

You're right! It happens when using RetrieveMultipleAsync(). It happens to me that after some minutes it gets cleaned-up (probably due to GC things...).

@Ivan-Colomer when calling retrieve multiple, Are you using RetrieveMultipleAsync or just RetrieveMultiple ?

Yes, I'm using RetrieveMultipleAsync(). Hope you can find where the issue is.

There is a somewhat active discussion that is going on this subject on the .net team as it relates to use of the underlying XML type buffer.    There was a fix checked into .net 7 that may impact this, ( we have not tested specifically for it yet ), See: https://github.com/dotnet/runtime/commit/d964b638f1ad47a729ab2e6cf4d6822f76ea3e4f
Can you recompile for .net 7 ( the GA version ) and see if you see this issue persisting?

Thank you, I have now tested with 7.0.100. The issue remains identical.
Maybe my test was insufficient? I created a new .Net7 console application, referenced the latest nuget package for DataServiceClient and tested RetrieveMulitpleAsync.

the issue fix was in the framework itself, so it should have been picked up..
thanks for the validation of what we were seeing here.
We will continue to poke at this and see if we can lock it down.

Ok, Update on this.. We think we have finally got this locked down,  the .net 6 codebase made it a bit easier to spot.
The next update for the Dataverse Service Client should resolve this, and we will close this bug with that update.


wg_engine: ++infra

see: https://github.com/gfx-rs/wgpu-native/releases

as an alternative we ca use folowing code
    webgpu_dir = meson.global_source_root() + '/webgpu_native'
    wgpu_dep = meson.get_compiler('cpp').find_library('wgpu_native', dirs: [webgpu_dir], required: false)
    engine_dep += [declare_dependency(
        dependencies        : wgpu_dep,
        include_directories : [
            include_directories('.'),
            include_directories('webgpu_dir')
        ],
        sources : source_file
    )]

@SergeyLebedkin please make a better way with your PR. thanks.


Errors Overhaul

For some historic reasons (mistakes were made), Finch had RequestErrors error case embedding a sequence of throwables for the sake of error accumulation in product endpoints. That type is rather confusing and not easy to work with given that it sends a wrong message to the user implying that errors could be nested indefinitely, which is obviously not true (we flatten them while collecting). That promoted some non-trivial error handling solutions that has nothing to do with how multiple errors are actually represented.
To eliminate the gap between how things are looking at runtime and their actual type, this PR removes RequestErrors (Multiple in this case) and replaces it with the new error type Errors that embeds a non-empty list of Finch errors. This the new type indicates that the accumulated Errors are always flattened and only contain Finch's own errors (not arbitrary exceptions).
Circe does it the same way so it must be smart (c).
While the previous API is still supported (the change is source compatible), the behavior had to be changed a little. Previously, we accumulated all exceptions in the product endpoint (the RequestErrors type allowed it). Now, we're only accumulating Finch's own errors (parse errors, type conversion errors, missing entities errors) and fails-fast with the fist arbitrary exceptions.
I personally think the new behavior is more valid and safe. When an arbitrary failure occurs, it's not super clear that's safe to evaluate the next endpoint in the product so failing fast seems a reasonable way to deal with it. When a Finch's own error occurs, it's quite an obvious thing to do to keep accumulating b/c it's known those errors are local to a given request and should not affect an overall system.

Current coverage is 77.03% (diff: 76.92%)

Merging #694 into master will decrease coverage by 2.59%

@@             master       #694   diff @@
==========================================
  Files            30         30
  Lines           643        640     -3
  Methods         622        620     -2
  Messages          0          0
  Branches         21         20     -1
==========================================
- Hits            512        493    -19
- Misses          131        147    +16
  Partials          0          0

Powered by Codecov. Last update 36dc12c...18129b5


Any way to use babel-eslint here?

Nice project! How we can use babel-eslint?

There's no way to use a custom parser with standard, sorry. I might reconsider if there's significant demand, but it's not likely.

I'm interested in this as well

I am currently using some ES7 features (such as await/async) which are not currently supported by ESPree.
It is annoying because these parse failures (unexpected token) break ESLint and prevent the detection of potential errors.
I am not sure what would be the best way to tackle this issue without breaking the various editor plugins that integrate standard.

@julien-f If Espree can't parse a token it will break, there's little that can be done about that.
I think the solution is to make sure Espree can parse all tokens; you could either make it parse the build output (yuck) or help Espree forward by creating an issue and / or a PR to add async / await support.

Extracting the core (#87) and the EsLint config (#132) would also helps in creating a fork using babel-eslint.

Re-opening for discussion.

I created babel-standard based on eslint-standard-config and babel-eslint.
@feross I added you as a collaborator (and @sebmck, Babel creator), feel free to play with it.

Is this line the only thing you changed?

@julien-f Why didn't you just fork this repo so history was retained? Unsure how (or if) you're planning on keeping it in sync.

I use a fork (https://github.com/bendrucker/standard/tree/babel) and it's indeed a one liner just to set the parser in ./rc/.eslintrc: https://github.com/feross/standard/compare/master...bendrucker:babel

@feross Yep.
@sebmck I just put things together for testing, but you're right I should have done so and will fix it.

@bendrucker may be you want to test npm scopes with your fork?

@ColCh Yeah I should have, but at this point you can just use Julien's.
I think it's reasonable to treat parsers as the one case where eslint config can be provided without defeating the purpose of standard. There's really only two (esprima-fb and babel-eslint). Babel's targeting future ES features so this is an ongoing concern.
This is also blocked by eslint as far as I can tell. Parsers are read from the config file and there's no way to provide them via CLIEngine.

I got it working today with 'eslint-config-standard' - thank you for that!!

What's the story around this. I too would love to be able to use async-await. Should I use babel-standard?

@jladuval You can use babel-standard, if you find some bugs please report them there.

The latest version of standard (3.12.0) now supports custom JS parsers!
To use a custom parser, install it from npm (example: npm install babel-eslint) and add this to your package.json:
{
  "standard": {
    "parser": "babel-eslint"
  }
}

That's it. Enjoy!

Using the latest version here with config and I still get unexpected token on '=>'

I am also experiencing troubles using custom linters in Atom's lint engine. Not sure if it is standard's problem or linter-js-standard's.


Iframe fullscreen logging

Changes to add logging to the fullscreen iframe button. I'm not sure what we should use as the log action but what I added seems right.
I think we might want to add something to the logStack that includes the path for the iframe? Maybe also the path to the originating page although that can be inferred from the ids we are attaching.
Also I wasn't sure where to add this information into this document. I was thinking it doesn't fit in any existing category since iframes can be a part of any app (record and recordset. even the modal popups in recordedit).

Let's discuss this in a meeting with @hongsudt. Depending on whether we want to capture the stack and/or add another level to it, you would have to make some modifications to this implementation. The current implementation is completely ignoring stack, stack-path, and app-mode. So regardless of where users clicked on the button the action will always be :;iframe;fullscreen. Also the last word in the action string is always a verb, so we need to modify the text as well.

@RFSH this has been updated to include a log stack that communicates what page the action came from. I updated the document as well at the very bottom with it's own section. I wasn't sure where else to put it since iframes can be in any recordset view in any popup.


Vultr documentation for cloud-config-url

Hi,
it is not clear when and how should be "cloud-config-url" set. Could anyone expand on that? For example, what address should I provide? what it should point to exactly? what would be an example of output? what would change if the option is left invalid?
Thanks

I submitted a ticket to vultr about this recently and it seems they still don't have cloud-config support in the control panel. The simplest way for me is to put my cloud-init in a gist and replace the cloud-config-url with the "raw" file url.

That said, no matter what I do the partitioning comes out wrong, with the root partition only receiving 373mb, making the whole endeavor worthless so far.

Oh I see, those instructions are for installing to RAM. This blog post explains how to install to disk.


Upgrade electron-builder

Before submitting, please confirm you've

[x] read and understood our Contributing Guidelines
[x] completed Mattermost Contributor Agreement
[x] executed npm run lint:js for proper code formatting

Please provide the following information:
Summary
Upgrade electron-builder.
It has an issue about Linux icon. As a workaround, macOS icon is also used for Linux in this PR.
Issue link
#702
Test Cases
Download artifacts. Then, check artifacts name, package name, publisher name and icons.
Additional Notes
https://circleci.com/gh/yuya-oc/desktop/736#artifacts

@yuya-oc Looks like only the Zip packages were uploaded on CircleCI:

@jasonblais Thanks to let me know. Artifacts name had been changed by electron-builder. Here is fixed version.
https://circleci.com/gh/yuya-oc/desktop/736#artifacts

@yuya-oc I'm getting an error when trying to launch the app (Mattermost 24 is equivalent to Mattermost)

@jasonblais Reproduced also on local build. Probably it's a bug of electron-builder. (similar issue: https://github.com/electron-userland/electron-builder/issues/784)
But possibly it's good time to move to zip (it works). And it's required for auto-updater.

@yuya-oc Will there be any side effects to the user?
Were we able to use the .dmg files for auto-updates, I forget? I think you mentioned the flow for .dmg files would be similar to Windows .zip files? https://github.com/mattermost/desktop/pull/588

@jasonblais I think there are nothing other than filename (of course we should test though).
For mac, dmg and zip are almost same, but they have different purpose in auto-updater context.

dmg: For initial installation.
zip: Downloaded and applied by updater. It also can be used manually as well as our tar.gz.

@yuya-oc So would auto-updates work for .dmg? Or would the user download .dmg initially (if desired), but then .zip files for updates?

As for using tar.gz or zip files, I'm okay moving to .zip files if there are no changes to the user. Especially since we'll be changing to it anyways soon.

auto-updater would work for both of zip and dmg. But they check whether .zip exists on the server. So their difference is just file format (and some extra pictures for appearance of dmg). I'll prepare zip version.

https://circleci.com/gh/yuya-oc/desktop/740#artifacts

Tested on Ubuntu 16.04. No issue found, and icons are correctly included.

Thanks all! @yuya-oc we should be good to merge this one.


When i add a new layer to VSSM, the training process has the "Parameter indices" error?

I want to add a new network layer to class VSSM(nn.Module), like following:

But when training the network, which has the following error:

Could you give me some advice to solve the problem, thank you!

Please ensure that all the parameters are involved when calculating loss, otherwise, some parameters may not recieve gradients.


Migration from Mattermost is fail

I'm trying to migrate from mattermost to zulip.
when I tested with mattermost docker container, It worked.
But when I tried to migrate 50GB real data, It fails.
The error log says about duplicate key.
I don't know what makes it fails.
Is there anyone knows about this?

Please drop by the #production help channel in the Zulip development community to get help with your question, or email<EMAIL_ADDRESS>You will need to provide the error log output to get help.


Error in Microsoft Edge and IE 11

Hi,
I am using the latest Nuxt with your nice module. I have noticed that since then I have an error within our "beloved" browsers like Edge and IE11.
Screen: https://i.imgur.com/RI2eJYn.png
Error message: SCRIPT1028: SCRIPT1028: Expected identifier, string or number
You can verify the errors also here in staging: https://timework.netlify.com
If by any means I am mistaken and it has nothing to do with your module, I apologize.
Thank you.

Thank you for your issue! I'll be able to take a look at this in the next couple of days.

Thank you for your time!

Might be useful later: https://github.com/visualfanatic/vue-svg-loader/issues/63

Just ran into the same issue, any ideas to workaround this in the meantime?
Regards

@simllll I'll get to coding during the weekend. I have to find a solution within a week so I will keep you posted.

I'm sorry I've been so busy guys. I'll happily review and accept any PR's @ladislavsulc, thank you for very much for your help!

So far as I can say the origin of this issue lies in https://github.com/visualfanatic/svg-to-vue
https://github.com/visualfanatic/svg-to-vue/blob/1232c45a302f5a654be0d6444aeca15492e03d4f/index.js#L82
as they have changed the export, iexplorer cannot handle it. and it does not get babbled.
If you look in the master branch of the svg-to-vue though, it seems they have completly rewritten this part already (https://github.com/visualfanatic/svg-to-vue/blob/master/index.js). Maybe it works with the current code already, but it's just not published yet?
regards

Sorry guys, I am more a designer than a JS dev so I will have to leave this to the more skilled people. In the meantime, I will have to go with https://github.com/Developmint/nuxt-svg-loader since I need to support IE11 and Edge as well.

@simllll and I got it running with a little workaround: We have added babel-loader on client, which seems to do the job. I also created a PR with this fix included (#10). Hopefully this also works for others.

Closed with #10 and released in v0.1.6.


CLI tool plan?

Hi Matt, Thanks for great libarary !
Is it possible to add a cli to wrap it to command line interface?

You're quick! Thanks for the interest.
There is already a command line tool for this, actually. :) That's what Caddy is.


fix lint issues

Thank you for your contributions. Can you please try to make all tests pass?

Yet there are all builds passing!

Thanks.

No Problem!

If you don't mind I'll add you to the list of contributors in package.json.

Yes, please!


Rich Text Editor

There are currently many areas in which free text can be written, but without formatting functions. The idea would be to establish a simple rich text editor in selected places, not necessarily in every free text field. This would allow researchers to improve their descriptions with simple formatting options. Enumerations, lists, tables and the insertion of graphics would also be important here.

This is one of the aspects where the final goal of damap needs to be set/considered.
Originally we intended to eventually shift into machine readable exports (JSON) only, which removes the need for this feature.
If the expectation is instead to stay with the current product then it makes a lot of sense. in this case the tempalte export might be problematic, if it does not support the formatting. Also the character limitation of 4k chars for most fields would be an issue for formatted text (and is otherwise a boon for machine readadable exports).


question regarding diffusion model

Hi, great work this is exciting.
I have some questions regarding the diffusion model.
1)i noticed that step_i is not used in noising or denoising, as done in standard diffusion models. So usually, step_i is used with a noise scheduler to determine amount of noise to add. Is there a reason for this?
2)denoising is done when denoised image reaches target image. I understand breaking, but breaking the denoising process... is this a common thing?
I'm confused if we can call this a diffusion model... or it seems more of an iterative transformer decoder?

Thank you! I'm glad you find this work exciting :)

The model is given both the current render and the target render. Therefore, the model doesn't really need to know step_i, since that should be derivable from the current and the target. This makes sampling a lot easier as the noise level is estimated by the model.

If we just ran the denoising process, we would get a distribution of programs that could satisfy the target. Since our evals our fornal, where we have to match the image almost exactly, we stop. In fact, we can even do tree search on this model.

The method presented here is a discrete diffusion model (kind of like DiGress), where the transformer decoder is just modeling the discrete jumps using an auto regressive approach. But we do add discrete Markovian noise, and we do have a model that learns to undo this noise.


Support non-source-tree builds via autogen.sh

Hello,
currently autogen.sh does not support non-source-tree builds. The Gnome Common Migration page shows a nice autogen.sh template that can be used as a starting point to support non-source-tree builds. It would be nice if you could add support for this.

For me, the typical out-of-source build is a build from tarball (./configure, make, make check, make install). It succeeds. So I wonder what exactly is your problem, especially when you say 'autogen.sh' is somehow involved.

I am trying to build using the GNOME build tool called jhbuild. It can automatically checkout from git and perform a complete build. However with jhbuild I can only specify that the project uses autotools. It will then invoke autogen.sh for me and expects that autogen.sh calls configure.

The 'normal' pattern is to call 'autogen.sh', 'bootstrap' or 'buildconf' and thereafter ./configure.
This is the same for all projects that I regularly work on (or build as dependency):
libunistring, libidn, libidn2, nettle, gnutls, libmicrohttpd, libmetalink, curl, wget, wget2, ...
So there must be something wrong with or missing in jhbuild or (maybe more likely) in your configuration. Or jhbuild is simply the wrong tool for what you are trying to do !?

I took a look at the jhbuild docs - they should add another step for a separate ./configure run. That would make the build of standard projects much easier.
If you have more info that might change anything, feel free to reopen.


Write out the "supplier" to SBOM formats

Originating from this comment, the NTIA minimum elements calls out for supplier information, and both CycloneDX and SPDX provide according fields. However, there is no strict definition of that the "supplier" of an Open Source package should be. In particular, the "supplier" should not be the hosting website of a package.
That said, I'm proposing the following logic to deduce the supplier (try in order, first match wins):

For package managers that support it, use the organization name or similar (see e.g. Maven).
Use the namespace of a package if it contains "com" or "org" and is not a recognized VcsHost.
Use the author / developer if there's only a single one; prefer its organization (if supported by the package manager, see e.g. Maven) over its name.

Implementing this probably requires to introduce a dedicated "organization" field to the Package model; currently, e.g. the Maven-provided organization is simply added to the set of authors.

As discussed in ORT development meeting of September 4th, let make it possible to curate the organization field.

For reference below an example of SPDX with supplier info generated with Syft
{
 "spdxVersion": "SPDX-2.3",
 "dataLicense": "CC0-1.0",
 "SPDXID": "SPDXRef-DOCUMENT",
 "name": "/opt/python",
 "documentNamespace": "https://anchore.com/syft/dir/opt/python-2e170c31-f61e-4077-aeb5-e6ec88ef974e",
 "creationInfo": {
  "licenseListVersion": "3.21",
  "creators": [
   "Organization: Anchore, Inc",
   "Tool: syft-0.89.0"
  ],
  "created": "2023-09-03T14:15:13Z"
 },
"packages": [
  {
   "name": "CacheControl",
   "SPDXID": "SPDXRef-Package-python-CacheControl-b45cb057d32f67f5",
   "versionInfo": "0.13.1",
   "supplier": "Person: Eric Larson<EMAIL_ADDRESS>Frost Ming<EMAIL_ADDRESS>William Woodruff<EMAIL_ADDRESS>   "originator": "Person: Eric Larson<EMAIL_ADDRESS>Frost Ming<EMAIL_ADDRESS>William Woodruff<EMAIL_ADDRESS>   "downloadLocation": "NOASSERTION",
   "filesAnalyzed": false,
   "sourceInfo": "acquired package info from installed python package manifest file: /versions/3.10.13/lib/python3.10/site-packages/cachecontrol-0.13.1.dist-info/METADATA, /versions/3.10.13/lib/python3.10/site-packages/cachecontrol-0.13.1.dist-info/RECORD",
   "licenseConcluded": "NOASSERTION",
   "licenseDeclared": "NOASSERTION",
   "copyrightText": "NOASSERTION",
   "externalRefs": [
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:eric_larson_\\\u003cericlarson_project:python-CacheControl:0.13.1:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:eric_larson_\\\u003cericlarson_project:python_CacheControl:0.13.1:*:*:*:*:*:*:*"
    },
    {
     "referenceCategory": "SECURITY",
     "referenceType": "cpe23Type",
     "referenceLocator": "cpe:2.3:a:eric_larson_\\\u003cericlarsonproject:python-CacheControl:0.13.1:*:*:*:*:*:*:*"
    },


[Feature request] Optional automatic genome size estimation

Hi George,
I wanted to discuss the option of adding an 'auto' mode to the -c parameter.
In my opinion, the assembly of novel prokaryotes or at least the assembly of genomes that will be taxonomically classified AFTER they have been assembled, is a common use case of assemblers in general.
Therefore, it doesn't surprise me, that both shovill and dragonflye contain a genome size estimation step.
Both shovill and dragonflye use KMC for genome size estimation based on k-mer counting (see Line 136 in shovill and Line 214 in dragonflye). Both use the basic but well known formula to estimate the genome size:
For a given sequence of length L,  and a k-mer size of k, the total k-mer’s possible will be given by ( L – k ) + 1 (see this paper or for example this tutorial).
A way to implement this in snakemake would be something like this:
# Since kmc is unable to deal with tabs in the fastq headers (see https://github.com/refresh-bio/KMC/issues/42#issuecomment-340614580),
# we need to sanitize the input reads before running kmc. So here we replace tabs with underscores. Maybe hybracter does this already
rule sanitizeInputReads:
    input:
        longReads = lookup(
            query="sampleID == '{sampleID}'",
            within=sampleDF,
            cols="longReadFastq",
        ),
    output:
        longReads = temp(os.path.join(INTERIMPATH, "{sampleID}", "sanitized", "{sampleID}_sanitized.fastq.gz")),
    conda:
        os.path.join(workflow.basedir, "envs", "assembly.yaml")
    shell:
        """
        seqkit replace -p "\t" -r '_' {input.longReads} -o {output.longReads}
        """

# this is the actual rule that runs kmc. It needs to be defined as a checkpoint, so that the helper function can access the output easily
checkpoint kmc:
    input:
        longReads = rules.sanitizeInputReads.output.longReads,
    output:
        kmcLOG = os.path.join(RESULTPATH,"{sampleID}", "genomeSize", "{sampleID}_kmcLOG.txt"),
        dir = temp(directory(os.path.join(RESULTPATH,"{sampleID}", "genomeSize"))),
    shadow: "shallow"
    conda:
        os.path.join(workflow.basedir, "envs", "assembly.yaml")
    shell:
        "kmc -k21 -ci10 -v -fq {input.longReads} {wildcards.sampleID} {output.dir} > {output.kmcLOG} 2>&1"

# This is the helper function that opens the kmc log file and reads the number of unique k-mers
# For a given sequence of length L,  and a k-mer size of k, the total k-mer’s possible will be given by ( L – k ) + 1.
# This can be used to estimate the genome size by counting the number of unique k-mers in the sample.
# Since the genome length is much bigger than the k-mer size (L >> 21), the genome size can be estimated as the number of unique k-mers.
# Since the the estimated genome size is used to check if the chromosome assembly is complete or not, we use 80% of the estimated genome size to be on the safe side. This could obviously be adjusted.
def get_kmers(sampleID):
    with checkpoints.kmc.get(sampleID=sampleID).output.kmcLOG.open() as file:
        for line in file:
            if "No. of unique counted k-mers" in line:
                return int(float(re.search(r"No. of unique counted k-mers\s*:\s*([\d\.eE+-]+)", line).group(1)) * 0.8)

# Yes, I am running a snakemake/snaketool workflow within a snakemake workflow. Here I just want to use it as an example on how to use the helper function (see params: chromosomeSize)
rule hybracter_long:
    input:
        longReads=lookup(
                query="sampleID == '{sampleID}' & group == 'ONTonly'",
                within=sampleDF,
                cols="longReadFastq",
        ),
# this is added, so that kmc is run before this rule
        kmc= lambda wildcards: checkpoints.kmc.get(sampleID=wildcards.sampleID).output.kmcLOG,
    output:
        dir=directory(os.path.join(RESULTPATH, "{sampleID}", "hybracter_long")),
        summary=os.path.join(RESULTPATH, "{sampleID}", "hybracter_long", "FINAL_OUTPUT", "hybracter_summary.tsv"),
    threads: workflow.cores * 1
    log:
        os.path.join(LOGPATH, "{sampleID}", "logs", "{sampleID}_hybracter_long.log"),
    conda:
        os.path.join(workflow.basedir, "envs", "assembly.yaml")
    params:
# this uses the helper function to get the genome size (or 80% of that number, see above)
        chromosomeSize = lambda wildcards: str(get_kmers(wildcards.sampleID)),
        plassemblerDatabase=config["tools"]["plassembler"]["dbpath"],
    shell:
        """
        hybracter long-single \
        -l {input.longReads} \
        -s {wildcards.sampleID} \
        -c {params.chromosomeSize} \
        -o {output.dir} \
        -t {threads} \
        --databases {params.plassemblerDatabase} \
        --no_medaka > {log} 2>&1
        """

I know and appreciate that you want to keep hybracter as simple and stable as possible, and adding an additional dependency like KMC would add complexity. However, I think that this could add a lot of value in terms of being able to automate the assembly process. Also, KMC is actively developed, so any bugs that could cause problems will also be fixed.
Best wishes,
Richard

Hi @richardstoeckl ,
This is a very reasonable request. I did some kmc experiments when I started hybracter, but the issue is more that the estimated genome size needs to account for plasmids - so I would probably minus e.g. 500000 bases off the estimate.
Give you have done 90% of the implementation here , I certainly can't say no! - hopefully this week I'll find some time to push an update with this and a couple of other things, including the archaea reorientation :)
George

@richardstoeckl ,
I had a look at this this arvo. Are you getting reasonable 21mer counts for your data? They are a massive overcount for me (even with pretty new Nanopore data). With R9 SUP I get 9.5M for s aureus (should be 2.8M), and with R10.4.1 I get 4.5M for e faecalis (should be 3M).
George

@richardstoeckl - your code is amazing. Seems to be working out of the box. I'll run it through the CI test suite and merge it - feel free to make a cosmetic PR if you want to be recognised on the repo as a contributor, I'll gladly merge it -  thanks again!
George

@richardstoeckl - your code is amazing. Seems to be working out of the box. I'll run it through the CI test suite and merge it - feel free to make a cosmetic PR if you want to be recognised on the repo as a contributor, I'll gladly merge it - thanks again!
George

That would be an honor! Glad to be able to help improve this amazing pipeline :)


Go live 5-26-21

PR 3495

Docs Build status updates of commit 1c1f3a4:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

msteams-platform/concepts/deploy-and-publish/apps-upload.md
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Score on the right

Hi,
Does the library delete the score on the right side of the scoreboard?

Thank you

It does, but only for 1.20.4+ clients. You can also set a custom value instead of a number.

I'm using 1.21 so it's great for me. Thank you!


starward: error in Out-File

It seems that this error causes the setup wizard to not be skipped
And also for some reason config.ini is a folder
This is the content of scoop\persist\starward

Here is the terminal output:
PS C:\Users\redacted> scoop install starward
Installing 'starward' (0.11.3) [64bit] from Small-Ku_turbo-bucket bucket
Starting download with aria2 ...
Download: Download Results:
Download: gid   |stat|avg speed  |path/URI
Download: ======+====+===========+=======================================================
Download: 3bdfdf|OK  |   1.0MiB/s|C:/Users/redacted/scoop/cache/starward#0.11.3#https_github.com_Scighost_Starward_releases_download_0.11.3_Starward_Portable_0.11.3_x64.7z
Download: Status Legend:
Download: (OK):download completed.
Checking hash of Starward_Portable_0.11.3_x64.7z ... ok.
Extracting Starward_Portable_0.11.3_x64.7z ... done.
Running pre_install script...
Out-File : Could not find a part of the path 'C:\Users\redacted\scoop\persist\starward\config.ini'.
At line:1 char:85
+ ... UserDataFolder=$persist_dir\data" | Out-File $persist_dir\config.ini}
+                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (:) [Out-File], DirectoryNotFoundException
    + FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.OutFileCommand

Linking ~\scoop\apps\starward\current => ~\scoop\apps\starward\0.11.3
Creating shortcut for Starward (Starward.exe)
Persisting data
Persisting config.ini
'starward' (0.11.3) was installed successfully!
Notes
-----
Setup wizard would be skipped because of preset value of user data by pre-install scripts.

Can you delete the folder and try to install again?
Please reopen the issue when it still exists.


ideamaker: init at 4.3.3

Description of changes
A slicer for 3D printers.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Adding yourself to the maintainers/maintainer-list.nix should be a separate commit

You can apply the formatting suggestions by running nix run nixpkgs#nixfmt-rfc-style -- pkgs/applications/misc/ideamaker/default.nix that way you will automatically comply with RFC 0166

Result of nixpkgs-review pr 309130 run on x86_64-linux 1

Despite the nixpkgs-review result I was able to successfully build and run ideamaker on x86_64-linux after allowing unsecure and unfree packages

With NIXPKGS_ALLOW_INSECURE=1 NIXPKGS_ALLOW_UNFREE=1 I get:
Result of nixpkgs-review pr 309130 run on x86_64-linux 1

1 package built:

ideamaker

The category hierarchy may still be used for packages that should be imported using an alternate callPackage, such as python3Packages.callPackage or libsForQt5.callPackage.

@lovesegfault I know you previously maintained this package before it was marked broken and removed. Would you mind reviewing?
After some sleuthing about with gdb and ghidra, I learned the segfault was due to the application needing and old (like 8 years old) version of curl and openssl.


New recents not working

Rom : Pixel Experience 8.1.0
Device : Yu Yuphoria ( lettuce )
Module : 1.0.0-116
Log : 05-12 17:40:50.093 I/Xposed  (19268): -----------------
05-12 17:40:50.093 I/Xposed  (19268): Starting Xposed version 90.2-beta3 (Systemless), compiled for SDK 27
05-12 17:40:50.093 I/Xposed  (19268): Device: YU5010 (YU), Android version 8.1.0 (SDK 27)
05-12 17:40:50.093 I/Xposed  (19268): ROM: OPM1.171019.018
05-12 17:40:50.093 I/Xposed  (19268): Build fingerprint: YU/YUPHORIA/YUPHORIA:5.1.1/LMY49J/YOG4PAS8A8:user/release-keys
05-12 17:40:50.093 I/Xposed  (19268): Platform: arm64-v8a, 64-bit binary, system server: yes
05-12 17:40:50.093 I/Xposed  (19268): SELinux enabled: yes, enforcing: no
05-12 17:40:52.912 I/Xposed  (19268): -----------------
05-12 17:40:52.912 I/Xposed  (19268): Added Xposed (/system/framework/XposedBridge.jar) to CLASSPATH
05-12 17:40:55.310 I/Xposed  (19268): Detected ART runtime
05-12 17:40:55.312 I/Xposed  (19268): Found Xposed class 'de/robv/android/xposed/XposedBridge', now initializing
05-12 17:40:55.609 I/Xposed  (19268): Loading modules from /data/app/xyz.paphonb.androidpify-xX8V3xtMy11jQ7DdwEdoyw==/base.apk
05-12 17:40:55.995 I/Xposed  (19268):   Loading class xyz.paphonb.androidpify.XposedHook
05-12 17:41:09.309 I/Xposed  (19631): [Android P-ify] [ERROR] : onAttachedToWindow: mIconManager = null

This feature is not supported.


Graph disconnected when using transfer learning

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): pip
TensorFlow version (use command below): v2.3.0-54-gfcc4b966f1 2.3.1
Python version: Python 3.8.5
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: (nvcc) 10.1, V10.1.243;
GPU model and memory: Nvidia 2080 Ti

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
When trying to obtain values for intermediete layers of a pre-trained model, I get a "Graph disconnected" error, even though it is connected. I can train it, evaluate it, predict images, etc.
Fragment of code (the whole code is in a Colab - below):
img_size = (299, 299)
IMG_SHAPE = (*img_size, 3)
base_model = tf.keras.applications.InceptionResNetV2(input_shape=IMG_SHAPE,
                                                    include_top=False,
                                                    weights='imagenet')
base_model.trainable = False
preprocess_input = tf.keras.applications.inception_resnet_v2.preprocess_input

data_augmentation = [
  tf.keras.layers.experimental.preprocessing.RandomFlip('horizontal_and_vertical'),
  tf.keras.layers.experimental.preprocessing.RandomRotation(0.2, fill_mode="constant")
]

data_augmentation = tf.keras.Sequential(data_augmentation)

global_average_layer = tf.keras.layers.GlobalAveragePooling2D()

top_layers = []
top_layers.extend([
    tf.keras.layers.Dense(512, activation="relu",
                          kernel_initializer="glorot_normal",
                          bias_initializer="glorot_uniform"),
    tf.keras.layers.Dropout(0.2)
])

prediction_layer = tf.keras.layers.Dense(1, activation="sigmoid",
                                         kernel_initializer="glorot_normal",
                                         bias_initializer="glorot_uniform")

inputs = tf.keras.Input(shape=(*img_size, 3))
x = data_augmentation(inputs)
x = preprocess_input(x)
x = base_model(x, training=False)
x = global_average_layer(x)
x = tf.keras.layers.Dropout(0.2)(x)
for layer in top_layers:
    x = layer(x)
outputs = prediction_layer(x)
model = tf.keras.Model(inputs, outputs)

(...)

model2 = tf.keras.Model(inputs=model.input,
                        outputs=model.get_layer('inception_resnet_v2').output)

Describe the expected behavior
Possibility to obtain values of intermediete layers of a pre-trained model - maybe possibility to see all intermediate layers in model.summary(), because currently it looks like this:
Model: "functional_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_2 (InputLayer)         [(None, 299, 299, 3)]     0
_________________________________________________________________
sequential (Sequential)      (None, 299, 299, 3)       0
_________________________________________________________________
tf_op_layer_RealDiv (TensorF [(None, 299, 299, 3)]     0
_________________________________________________________________
tf_op_layer_Sub (TensorFlowO [(None, 299, 299, 3)]     0
_________________________________________________________________
inception_resnet_v2 (Functio (None, 8, 8, 1536)        54336736
_________________________________________________________________
global_average_pooling2d (Gl (None, 1536)              0
_________________________________________________________________
dropout_1 (Dropout)          (None, 1536)              0
_________________________________________________________________
dense (Dense)                (None, 512)               786944
_________________________________________________________________
dropout (Dropout)            (None, 512)               0
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 513
=================================================================
Total params: 55,124,193
Trainable params: 787,457
Non-trainable params: 54,336,736
_________________________________________________________________

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
https://colab.research.google.com/drive/1cgwJho63-QtQYS3Tdk4O5jioDu_szYjl?usp=sharing
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

ValueError                                Traceback (most recent call last)
 in ()
1 model2 = tf.keras.Model(inputs=model.input,
----> 2                         outputs=model.get_layer('inception_resnet_v2').output)
5 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in new(cls, *args, **kwargs)
240       # Functional model
241       from tensorflow.python.keras.engine import functional  # pylint: disable=g-import-not-at-top
--> 242       return functional.Functional(*args, **kwargs)
243     else:
244       return super(Model, cls).new(cls, *args, **kwargs)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
455     self._self_setattr_tracking = False  # pylint: disable=protected-access
456     try:
--> 457       result = method(self, *args, **kwargs)
458     finally:
459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/functional.py in init(self, inputs, outputs, name, trainable)
113     #     'arguments during initialization. Got an unexpected argument:')
114     super(Functional, self).init(name=name, trainable=trainable)
--> 115     self._init_graph_network(inputs, outputs)
116
117   @trackable.no_automatic_dependency_tracking
/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
455     self._self_setattr_tracking = False  # pylint: disable=protected-access
456     try:
--> 457       result = method(self, *args, **kwargs)
458     finally:
459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/functional.py in _init_graph_network(self, inputs, outputs)
189     # Keep track of the network's nodes and layers.
190     nodes, nodes_by_depth, layers, _ = _map_graph_network(
--> 191         self.inputs, self.outputs)
192     self._network_nodes = nodes
193     self._nodes_by_depth = nodes_by_depth
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/functional.py in _map_graph_network(inputs, outputs)
929                              'The following previous layers '
930                              'were accessed without issue: ' +
--> 931                              str(layers_with_complete_input))
932         for x in nest.flatten(node.outputs):
933           computable_tensors.add(id(x))
ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_1:0", shape=(None, 299, 299, 3), dtype=float32) at layer "conv2d". The following previous layers were accessed without issue: []

UPDATE
So, when instead of
model2 = tf.keras.Model(inputs=model.input,
                        outputs=model.get_layer('inception_resnet_v2').output)

I use
model2 = tf.keras.Model(inputs=model.input,
                        outputs=model.get_layer('inception_resnet_v2').get_output_at(0))

Then it works! So I believe this issue can be closed now, but I think you should include it in the documentation (it is not included right now) - for this purpose, I am leaving this issue open.
All credit should go to Mitiku from this Stackoveflow thread. This is his answear:

The layer you are trying to use as output has two output nodes. The first connects the input of model2 to the output of the model2. The second output node connects the output of the model1 and the first layer of model2. By default, the layer output returns only the first output node. So what is happening is you are tying to connect the input of the model_merge(input of model1) with the first output node.
To following code shows this. Individual output nodes of the layer can be accessed using the get_output_at() method of the layer.
layer_output = model_merge.layers[-2].output # The first output node
layer_output_1 = model_merge.layers[-2].get_output_at(0) # The first output node
layer_output_2 = model_merge.layers[-2].get_output_at(1) # The second output node
Now the following two codes throws error because the graph is disconnected.
test2 = Model(inputs=model_merge.input, outputs=layer_output)
and
test2 = Model(inputs=model_merge.input, outputs=layer_output_1)
But the code below doesnot throw error, because the graph is connected.
test2 = Model(inputs=model_merge.input, outputs=layer_output_2)

Thanks for the issue. The docs are now updated.


Should we recommend SQLAlchemy NullPool?

I wonder if sqlalchemy-aurora-data-api should recommend using SQLAlchemy NullPool?
As I was figuring out https://github.com/cloud-utils/aurora-data-api/issues/43 I got looking at interactions between SQLAlchemy pooling, sqlalchemy-aurora-data-api and aurora-data-api.
The documentation for SQLAlchemy NullPool says:

A Pool which does not pool connections.
Instead it literally opens and closes the underlying DB-API connection per each connection open/close.
Reconnect-related functions such as recycle and connection invalidation are not supported by this Pool implementation, since no connections are held persistently.

Since aurora-data-api is fundamentally connection-less (part of what makes it great!), having SQLAlchemy make efforts managing connections seems like unneeded effort and, especially in failure situations, potentially confusing.

Thanks for the pointer, this sounds like a good idea for this package.

I'm actually not sure what the upside of using NullPool would be. While the overhead of creating a new aurora-data-api client is relatively low, it's not zero. I'd prefer to make aurora-data-api fully threadsafe instead of asking users to introduce more configuration that they may or may not apply.
AFAIK the only non-threadsafe portion of aurora-data-api right now is the transaction id bound to the connection. I think the transaction id will have to be made thread-local.

Filed https://github.com/cloud-utils/aurora-data-api/issues/47 to keep track of that.

I don't know of a reason that it's important to use NullPool. I just know that SQLAlchemy is doing work presuming that there is a very expensive TCP connection behind the database connector and that is not the case here. Why have SQLAlchemy doing work is does not need to do? How the SQLAlchemy connection management work balances against the fairly small, but not zero, cost of creating a new sqlalchemy-aurora-data-api instance? I don't know.


Declare license for font files

Type: Incomplete
Summary:
Declare license for font files
Details:
Font files missing declared license info
Resolution:
Update declared licenses according to readme.txt in each below font directory:
/www/resources/icons/crystal_project/* > LGPL-2.0
(see: https://web.archive.org/web/20110622084146/http://www.everaldo.com:80/crystal/?action=license)
(see: https://github.com/simplesamlphp/simplesamlphp/blob/92de42cb0a25ccbf46f78a0e2f3257272afb8ba2/www/resources/icons/crystal_project/readme.txt)
/www/resources/icons/experience/* > GPL-2.0
(see: https://web.archive.org/web/20100323174847/http://art.gnome.org:80/themes/icon)
(see: https://github.com/simplesamlphp/simplesamlphp/blob/92de42cb0a25ccbf46f78a0e2f3257272afb8ba2/www/resources/icons/experience/readme.txt)
/www/resources/icons/silk/* > CC-BY-2.5
(see: https://github.com/simplesamlphp/simplesamlphp/blob/92de42cb0a25ccbf46f78a0e2f3257272afb8ba2/www/resources/icons/silk/readme.txt)
(see: https://web.archive.org/web/20111210165148/http://famfamfam.com/lab/icons/silk/)
Affected definitions:

simplesamlphp 92de42cb0a25ccbf46f78a0e2f3257272afb8ba2

@bduranc, if you agree, I will update the license for the Crystal Project files since the link you provided is LGPL-2.1 instead of LGPL-2.0 based on section 6b ("Use a suitable shared...").

@capfei : Yes, please do.


ManagedIdentity and Workload Identity support

I really like what Aspire is doing and how it simplifies the development and deployment cycle. as I was looking through the issue for preview 3 I came across some issue that have considerations around connectionstrings.
in general I don't think that it is a good approach to have "full" connectionstrings in files like the manifest, though I understand that there might be circumstances where this might be preferable, I think we should have a better way of handling secrets espacially during deployment.
I am a big fan of Managed- and Workflow Identities and think it makes sense to use them when ever possible. there are some services that might not support ManagedIdentities and for this I would prefer to have them in a Keyvault and them have the connection to that KV be secured using a ManagedIdentity.
I think it would be great for Aspire to set in the manifest to use managed Identities where possible or where it is not possible a additional KV resource is added to the manifest and that in the manifest it ptells the tooling to put the credentials/connectionstrings into this KV and then use the secret from KV when added as a reference in another service.
the tooling (azd up, aspir8) could than create  the managed/workflow Identities, add the right role permissions and configure ACA or Aks to use that identity to connect to the services. this would make much safer production deployments and increase overall security.
I hope it is understandable what I am suggesting, sometimes things sound well formulated in my head, but they don't come out the way I'd like 😅
so please let me know if something is unclear.

We have plans on how to model secrets that are going to be fleshed out in the next release. Once we're done with the refactoring, they'll never show up in the manifest verbatim (unless you do it manually as part of WithEnvironment).

It's great that this is happening as of today already, but when I understand that correctly this is not part of the manifest.json, right? So probably this won't happen for Aspir8?
If it would be part of the manifest, we could just implement a "lite" version of this:
AddManagedIdentityRole(string roleId). That would make it easier to add permissions the ManagedIdentity that is created by azd up to existing resources. Most of the time, when you start a new project you often want to connect to existing Azure Resources. We have a lot shared Cognitive Search Resources for example, or when we connect to Azure OpenAI.

It's great that this is happening as of today already, but when I understand that correctly this is not part of the manifest.json, right? So probably this won't happen for Aspir8?

Aspir8 doesn't support azure at all. It's a pure kubernetes solution that has no dependency on azure resources.

If it would be part of the manifest, we could just implement a "lite" version of this:

We're pushing azure resources out of the manifest as first class concept. Instead we're moving to a single bicep resource to represent them. That'll open the door for modeling what you are trying to model.

@davidfowl

Today when you spin up an aspire app with azd, it creates a user assigned managed identity behind the scenes and manages various role assignments for them

I gave that a try and exported everything to biceps via azd infra synth.
My code looks like this right now:
var sqlServer = builder.AddSqlServer("sqlserver");

var db = sqlServer.AddDatabase("aspire_sample");
var db2 = sqlServer.AddDatabase("aspire_sample2");

var apiservice = builder.AddProject<Projects.AspireSample_ApiService>("apiservice")
    .WithReference(db);

builder.AddProject<Projects.AspireSample_Web>("webfrontend")
    .WithReference(db2)
    .WithReference(apiservice);

With that code I'd expect that aspire makes sure that only the ApiService has access to aspire_sample and only the Web project has access to "aspire_sample2".
The generated biceps code however generates one managed identity for all applications and assigns that as ADMIN to the sqlServer. I can't believe that this is best practice.
resource managedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: 'mi-${resourceToken}'
  ...
}
resource sqlserver 'Microsoft.Sql/servers@2022-05-01-preview' = {
  name: 'sqlserver-${resourceToken}'
  ...
  properties: {
    ...
    administrators: {
      administratorType: 'ActiveDirectory'
      azureADOnlyAuthentication: true
      login: managedIdentity.name
      principalType: 'User'
      sid: managedIdentity.properties.principalId
      tenantId: subscription().tenantId
    }
  }

In my opinion this generates multiple problems:

How do I assign myself, or an administrator group as admin when deploying?
Adding the managed identity as admin means if someone somehow manages to get access to the frontend projects and retrieve the managed identity, then they also have access to the whole sql server (and even to all other sqlservers in the same aspire project, too).

The correct way in my opinion is to create a managed identity for each service and assign it to the database. Even db_owner on the database would be better than having full admin access on every server.
Not sure if this is only an Azure SQL issue but it feels like a general issue to me.

The generated biceps code however generates one managed identity for all applications and assigns that as ADMIN to the sqlServer. I can't believe that this is best practice.

It's not. Right now the generated bicep hasn't been reviewed for best practices and I expect us to make big changes to the default model once we get the end-to-end working.

The correct way in my opinion is to create a managed identity for each service and assign it to the database. Even db_owner on the database would be better than having full admin access on every server.

How do you tell the application which identity to use in this case? In an app uses N services, do you pass it N managed identity client ids somehow and manually flow that to the SDK? Or am I missing something?
As an aside. I've been experimenting with a managed identity resource type based on the bicep resource primitive:
https://github.com/dotnet/aspire/tree/davidfowl/managed-identity

How do you tell the application which identity to use in this case? In an app uses N services, do you pass it N managed identity client ids somehow and manually flow that to the SDK? Or am I missing something?

Sorry. I think the word "service" was a bad choice. With service I meant that each Container App (project) for example should have one managed identity representing the container app.
For example with terraform we currently deploy a managed identity with each container app and then use some third party terraform module to assign the managed identity as db_owner to the database the app is allowed to access. Sadly currently no Azure API seems to support adding users or managed identities to databases. I hope this sometime gets fixed.

Sorry. I think the word "service" was a bad choice. With service I meant that each Container App (project) for example should have one managed identity representing the container app.

This seems like a reasonable default, still, before we make any changes here we'll review the default security model with our security gurus and decide what the "default" should be if nothing is specified. Then there's be a mode where you get full control and can decide what to do (like keeping your terraform model).

Spike here


Dehydrated does not keep track of deploy hook failures

In case a deploy hook fails, dehydrated does not make a note of this to try to redeploy the certificate on the next cron run.
Here's some sample output of this:
$ dehydrated -c
# INFO: Using main config file /etc/dehydrated/config
Processing certtest.itassistans.net
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for certtest.itassistans.net...
 + Responding to challenge for certtest.itassistans.net...
 + Challenge is valid!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
writing RSA key
Traceback (most recent call last):
  File "/usr/local/bin/deploy_nsx_sslvpn_certificate", line 127, in <module>
    for line in file('/etc/dehydated/nsx-edges'):
IOError: [Errno 2] No such file or directory: '/etc/dehydated/nsx-edges'
$ vi /usr/local/bin/deploy_nsx_sslvpn_certificate
$ dehydrated -c
# INFO: Using main config file /etc/dehydrated/config
Processing certtest.itassistans.net
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Jun 28 08:25:00 2017 GMT (Longer than 30 days). Skipping renew!

In this case the problem was a typo in my script :-) But it might just as well have been some kind of temporary retryable error causing the deploy to fail.
In order to fix this problem, dehydrated needs to keep a note of failed hook executions and re-try them on next cron run.

I feel that this assumes too much. For one, how often is the script executed through cron--and is it executed through cron at all? Some people run it through cron every day, some every week, others every two weeks... there's no guarantee that the "next cron run" will happen in time to deploy the cert before the previous cert expires.
The script can also be run (with hook) completely manually. In this case, waiting for the "next cron run" may never happen--and if the script is executed manually later, it's very questionable that the user would want the script to suddenly start redeploying scripts from the last run, especially if they were now using a different hook entirely. (And if you want a hook to incorporate this feature in a self-contained manner, you are always free to write one to fit your needs!)
Finally, the failed deploy may be already caught through manual review of cron logs, or by inspection of the installed certificate, and fixed through manual intervention. It would be pointless at best to redeploy in this situation.
For all of these reasons, I feel it is better not to automatically retry failed hook calls. Any such functionality would be better handled from within the hook itself, rather than assuming anything about how the main script will be run.

I hear what you're saying but I have to disagree, because leaving Dehydrated as it is would mean inconsistent behavior.
If I run Dehydrated, and for some reason, it isn't able to obtain a certificate, re-running Dehydrated will attempt to obtain the certificate again if re-run. The cause of the failure is immaterial - it could be anything from the fact that the cron job didn't run due to a power failure, or a network problem causing Dehydrated to be unable to reach the LetsEncrypt servers, or an outage at LetsEncrypt. The only thing that matters is that LetsEncrypt does not have an up-to-date certificate in the certificate store.
The fact that running Dehydrated will re-try failed operations is therefore not surprising to an administrator. The fact that deploys are not retried on subsequent runs of Dehydrated (no matter if run from cron, manually, or in some other way) is surprising in that context.
Running dehydrated on a regular basis is still what you should do anyway in order to mitigate errors on the issuance side. There are many reasons Dehydrated may be unable to retrieve a certificate at any given point in time - anything from a network problem, or an outage on either the server running Dehydrated or an outage at LetsEncrypt. Personally I'd suggest an interval between 1-7 days, to permit enough retries, but that's neither here nor there.
With all this considered, I can understand the point of view that introducing a feature for retries in Dehydrated may be an undesired increase in complexity. Perhaps, in view of that, the fact that Dehydrated does have a certificate deploy hook at all is the design mistake?

There's a fundamental difference between retrying failed renews and retrying failed deploys.
A failed renew is easily detected by seeing that the expiry date hasn't been updated. The script naturally sees this and attempts another renewal. There's no additional logic needed; this is the same process as any renewal, whether it failed previously or not.
This script is not built to do everything. It's built specifically to handle the renewals and the process around that. Hooks are made available to extend the main script's functionality; wrappers can be written to add even more. But the functionality of the hooks and/or wrappers are not dehydrated's concern. All dehydrated has to do is call the hooks.
I do not agree that retrying failed renews is inconsistent with not retrying failed deploys. One activity is dehydrated's main function; the other is an optional feature enabled by other scripts not under dehydrated's purview. Let those other scripts do their own error detection and recovery, if they wish.
I also disagree with the suggestion to simply remove the hook altogether. It's an optional feature offered to allow others a way to write in additional functionality. There are those who use that hook, and successfully. I'm not persuaded that removing an optional way to extend functionality solves any issues yet raised.

I agree with what @txr13 said, here is an idea on how to solve this yourself:
Instead of doing all your operations directly in the deploy hook write the parameters into a queue, then on the exit hook or even on the unchanged hook run a script that checks if there is anything in the queue, does whatever it needs to do, and removes it from the queue.
If it fails it will just try again on the next hook execution.
There are a lot of other options for you to implement this easily for yourself, and there might even be cases where it would be bad if a deployment would run twice (or unexpected), so I'm not going to implement this inside of dehydrated.


Update local election info - Cumberland County

Spreadsheet: https://docs.google.com/spreadsheets/d/1dk58MD0qBAM37BDOZwKIz10hp5AARAIX8u_qvuDnYaI/edit#gid=0
Instructions: https://github.com/OpenMaine/maineballot/wiki/Updating-Maine-Ballot-for-an-upcoming-election

Cumberland County finished out in the spreadsheet.  Town meetings NOT on June 8 are noted in the "June 2021 ballot" column. (Info not yet copied over to local.csv)


ts-repeat is a custom directive

My ng-repeat element is an attribute directive. Can it be that the table-sorting can't work? I get the following error:
Error: [$compile:ctreq] Controller 'browseContent', required by directive 'browseContentItem', can't be found!
http://errors.angularjs.org/1.4.3/$compile/ctreq?p0=browseContent&p1=browseContentItem
    at REGEX_STRING_REGEXP (http://localhost:3000/assets/javascripts/main.js:6298:779)
    at getControllers (http://localhost:3000/assets/javascripts/main.js:10120:255)
    at nodeLinkFn (http://localhost:3000/assets/javascripts/main.js:10137:162)
    at compositeLinkFn (http://localhost:3000/assets/javascripts/main.js:10035:964)
    at publicLinkFn (http://localhost:3000/assets/javascripts/main.js:10014:376)
    at link (http://localhost:3000/assets/javascripts/main.js:21093:45)
    at invokeLinkFn (http://localhost:3000/assets/javascripts/main.js:10223:317)
    at nodeLinkFn (http://localhost:3000/assets/javascripts/main.js:10137:70)
    at compositeLinkFn (http://localhost:3000/assets/javascripts/main.js:10035:964)
    at compositeLinkFn (http://localhost:3000/assets/javascripts/main.js:10035:1073) <tr browse-content-item="" ng-repeat="item in items" item="item" ng-class="{'odd': ng-class-odd, 'even': ng-class-even}" ts-repeat="">

It seems like you have some sort of dependency problem, but it is impossible for me to tell without being able to run your code. Can you reproduce it in a plunkr?

Hi @mattiash , I created my own sorting mixin. Next time I run into this issue, I'll try to recreate it in a Plunkr. Thanks for your response.


Pomodoro combo habit issue

I selected not to have pomodoro combos give me extra habit credit (under the Pomodoro Combo Habit section), yet the add-on keeps re-creating a habit for pomodoro combos (which adds unnecessary clutter to my habits list, since I'm not using this habit). This seems like it might be a bug, so I thought I'd report it.
Thank you for the very helpful add-on, by the way!

Hi, thanks for report , it does seem like a bug.
There might be a workaround for now, follow these steps:

Leave the Combo Habit unchecked
Check custom Habit for Combo and choose your regular Pomodoro Habit to be also for Combo.
Save and then delete the combo habit in Habitica.
Let me know if it works for you :)

@savne Hi, thanks for the report , it does seem like a bug.
There might be a workaround for now, follow these steps:

Leave the Combo Habit unchecked
Check custom Habit for Combo and choose your regular Pomodoro Habit to be also for Combo.
Save and then delete the combo habit in Habitica.
Let me know if it works for you :)

Thank you @ofekmiz! This workaround worked :).

Thank you @ofekmiz! This workaround worked :).


Deploy does not include directories on Windows

Directories in a function's directory are not included when apex deploy is run on Windows. So, requireing something in node_modules will fail, for example.
Deploying the same project on Linux does work.
In both setups Apex 0.6 is installed.

Hey, could you paste apex deploy --log-level debug output from both Windows and Linux (for the same project). I'm completely unfamiliar with Windows but we can try to fix this :)

Linux output:
• loading functions in functions
• loading function in functions/foo
• deploying 1 functions
• deploying                 function=foo
• creating build            function=foo
• add file to zip           file=index.js function=foo
• add file to zip           file=index.zip function=foo
• add file to zip           file=stringholder/foostring.js function=foo
• created build (762 B)     function=foo
• fetching config           function=foo
• config unchanged          function=foo
• code unchanged            function=foo
• cleaning 1 functions

Windows output:
   • loading functions in functions
   • loading function in functions\foo
   • deploying 1 functions
   • deploying                 function=foo
   • creating build            function=foo
   • add file to zip           file=index.js function=foo
   • add file to zip           file=index.zip function=foo
   • add file to zip           file=stringholder\foostring.js function=foo
   • created build (762 B)     function=foo
   • fetching config           function=foo
   • config unchanged          function=foo
   • code unchanged            function=foo
   • cleaning 1 functions

Hmm those look fine to me, producing the same byte size, is that the case with node_modules involved as well?

Yep, just verified with node_modules. Once again it's the same byte size, but Lambda is throwing the following after a deploy from Windows:
{
  "errorMessage": "Cannot find module 'async'",
  "errorType": "Error",
  "stackTrace": [
    "Function.Module._resolveFilename (module.js:338:15)",
    "Function.Module._load (module.js:280:25)",
    "Module.require (module.js:364:17)",
    "require (module.js:380:17)",
    "Object.<anonymous> (/var/task/index.js:2:13)",
    "Module._compile (module.js:456:26)",
    "Object.Module._extensions..js (module.js:474:10)",
    "Module.load (module.js:356:32)",
    "Function.Module._load (module.js:312:12)",
    "Module.require (module.js:364:17)"
  ]
}

Works fine after a Linux deploy.

I suppose the archive would literally have \\ in the path names, I've never really thought about that haha, haven't touched windows in ages, may have to switch that to a path.Join

@5loop could you use apex build foo > out.zip and check files paths inside zip?

The file paths in the .zip from apex build foo > out.zip look fine. However, if I upload that file manually or via aws-cli, I get the same error as above when invoking.
If I zip the files manually and then upload the .zip from that, there is no error.
Maybe it is a \\ vs / issue, like @tj mentioned. I know that Windows does some automagic file path stuff so that it can read files from Linux machines, which might be why everything looks fine inside of the .zip even though the paths are still breaking


Small cleanup and use ScalingMode instead of manual calculation

The black background is added for clarity only, it is not in the pull request.
Screencast From 2024-10-15 15-05-40.webm
Tried to keep the commit history clean.

I recommend looking at the changes separately for each commit. For some reason, the combined diff looks larger than it actually is.

These changes are really nice and I would definitely merge them if this were project weren't for a tutorial. I'm hoping for ALL of the versions of the code on this repository to be the same as the video so I'll probably just keep the repo the way it is.


Bad Framerate for "Hello World" Project on Linux

The following main.dart
import 'package:flame/game.dart';
import 'package:flutter/material.dart';
import 'package:mygame/mygame.dart';

void main() {
  runApp(GameWidget(game: MyGame()));
}

and mygame.dart
import 'package:flame/components.dart';
import 'package:flame/game.dart';

class MyGame extends FlameGame {
  @override
  Future<void> onLoad() async {
    add(FpsTextComponent(position: Vector2(0, 0)));
  }
}

only renders with about 20 FPS on my WQHD (2.560 x 1.440) Screen for both Debug and Release Builds on Linux. I'm using an NVIDIA GTX 1080 together with an i7 and 32GB of Memory. Whats interesting: If i lower the resolution (something like half of screen width/height) It goes up to around 59/60 FPS. The weird thing is that neither system monitors show a high load in cpu/gpu or memory nor does the flutter performance view show anything special.

Flutter doctor output
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.3.4, on openSUSE Tumbleweed 5.19.13-1-default, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Chrome - develop for the web
[✓] Linux toolchain - develop for Linux desktop
[✓] Android Studio (version 2021.1)
[✓] Android Studio
[✓] VS Code (version 1.71.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

• No issues found!

pubspec.yaml
name: mygame
description: A new Flutter project.
publish_to: 'none'
version: 1.0.0+1
environment:
  sdk: '>=2.18.2 <3.0.0'
dependencies:
  flutter:
    sdk: flutter
  cupertino_icons: ^1.0.2
  flame: ^1.4.0
dev_dependencies:
  flutter_test:
    sdk: flutter
  flutter_lints: ^2.0.0
flutter:
  uses-material-design: true
  assets:
    - assets/images/

If I had to guess I would say that there might be some kind of resource limitation on linux to the skia/canvas but as mentioned this is only a vague guess.

I tried this too and I get around 40 FPS.
I don't think this has anything to do with Flame though, but just like you say it has something to do with one of the underlying layers.

Is there any limitation on fps? I assumed that my PC should even be capable of handling around 120fps. I also saw that the same project renders with 90fps on my android phone.
Anyway I also support your idea of this being an issue with some "underlying technology" (eventually something within flutter itself). I would really like to get to the bottom of that - Any hints on what I could do to further investigate this issue?

There is no limitation on the fps, more than what the hardware reports that it can handle.

Any hints on what I could do to further investigate this issue?

I would have a look at the Flutter issue tracker and see if there is anything related to this on there. If there is not, I would try to create a minimal reproducible example that doesn't use Flame and just uses the canvas and post that as an issue on Flutter. (you can link this issue)