Memory leak in TF 2.3.1 when using Multi-worker training

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Tensorflow 2.3.1 container
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): Tensorflow 2.3.1 container
TensorFlow version (use command below): Tensorflow 2.3.1 container
Python version: Tensorflow 2.3.1 container

Describe the current behaviour
When using MultiWorkerMirroredStrategy, there is a memory leak happening during training. The memory leak is not present when only one node is used.
Here is the memory consumption of a single node trainning

Here is the memory consumption of multi-node training (1 chief and 1 worker)

The code is identical for both cases. The only thing that changes is the TF_CONFIG which is provided be GCP AI Platform.
Describe the expected behaviour
The memory consumption should not be increasing by time (epochs).
Standalone code to reproduce the issue
The code is available at this repo.

Maybe related to #42616

@kalosisz
Can you please try with TF nightly version and see if the issue still persists.Thanks!

@ravikyram
I have changed the order of operations on the dataset to cache, shuffle, repeat, batch, prefetch. I have re-run the code on 2.3.1 and nightly (see the repo).
The leak is even bigger for nightly. Here are the memory footprints of nightly (master and worker, resp.)

The same under 2.3.1 (master and worker, resp)

Hi @kalosisz, I took a look at the repo you have provided. I want to try and reproduce this on AI Platform Training. Can you provide the values you used for the various flags (eg learning rate, laters) as well as the data?
A few other things that could help to isolate the problem: can you call model.fit but without validation data and let me know what you see? Also you're running this with CPU only machines correct? Is there still a leak if you use two GPUs instead?

Hi @nikitamaia,
Yes, I'm using CPU only. The machines are n1-standard-4.
The data is available here gs://szilard-ml-sandbox-eu-public/random_data_tfrecords. The bucket is located in the EU and requester pays is switched on.
I've tried to remove the validation, but it did not help (run on nightly). The parameters I've run all the trials are --layer-size 100 10 --learning-rate 0.01 --epochs 50 --data-base-path gs://szilard-ml-sandbox-eu-public/random_data_tfrecords --training-examples 2799589 --validation-examples 350198 --evaluation-examples 350213

I haven't tried to run thin with GPU, mainly because it'd be a bit costly for me.

Because the requester pays param is on it seems I need the name of your project in order to access the data. I'm seeing the error message Bucket is requester pays bucket but no user project provided.

According to the documentation, you should use the -u option with you project. Please see link
Please let me know if it helped.

Thanks, that worked. I was able to access the data. However, when trying to train on AI platform and locally, I am getting the error UnboundLocalError: local variable 'logs' referenced before assignment
Here is the command I ran
!gcloud ai-platform jobs submit training TF_issue_44688 --region us-east1 \
    --package-path multi-worker/trainer --module-name trainer.task \
    --config config.yaml --job-dir $JOB_DIR -- --layer-size 100 10 \
    --learning-rate 0.01 --epochs 50 \
    --data-base-path $DATA_DIR --training-examples 2799589 \
    --validation-examples 350198 --evaluation-examples 350213

hey @nikitamaia, any updates on the matter? :slightly_smiling_face:

I was able to repro an increase in memory on AI Platform and also on GCE VMs. Unfortunately, no big updates after this initial investigation. But I will definitely keep this thread updated.

Thanks for the info. Out of curiosity, did the issue arrise in case of GPUs as well?

I met the similar problem in my project.
In my project,  the reason may be tf.Dataset.repeat(epochs) which will increase the memory until it grows to the peak of memory (which base on the epochs).  However, it's sad to me that my model require a minimum of 5 epochs and the memory only can support epochs <= 3.
you can try to set a small epochs in your data.py

@zhenql my exmaple is here to demonstrate the leak it does not how real world implication, however it might just have.
certainly, the number of epoch is not an issue, if not trained in distributed fashion. should the MultiWorkerMirroredStrategy be not considered experimental, these issue musn't persist.

@kalosisz well, my project is also trained with MultiWorkerMirroredStrategy, and the memory leak is relevant to tf.Dataset.repeat(epochs) in the distributed environment (certainly, single machine can not see the memory leak).  I just can located the memory leak might be here of my project.
I'm not sure whether your problem is related to repeat() or not but you can have a try if like.

Hey @nikitamaia, has there been any update from TF side? The error still persists in v2.4.0, even though the MultiWorker is apparently stable. Thanks!

Hey @nikitamaia, has there been any update from TF side? The error still persists in v2.4.0, even though the MultiWorker is apparently stable. Thanks!

Closing for no activity


Note Colors menu scrolls down to 9

Describe your bug here.
https://user-images.githubusercontent.com/45212377/144760207-ad6c28a8-8c56-4fe5-85b1-f30d29480f78.mp4
another bug that I simply felt like reporting
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Source
What is your build target?
Linux
Did you edit anything in this build? If so, mention or summarize your changes.
nothing

will fix

I couldn't recreate this bug on Windows/HTML5.

Does it still happen in the newest source build?

Does it still happen in the newest source build?

@ShadowMario
yes, I only tested that on linux since I don't own a Mac myself, or have a Windows PC laying around, but as far as I can tell, this bug doesn't exist on Windows

Did you use any external program to open the .exe file or you compiled it yourself?

Did you use any external program to open the .exe file or you compiled it yourself?

I compiled it myself, Haxel version is 4.2.4 (latest at the time)

@ShadowMario he is using extra keys maybe

If he's using extra keys then i have nothing to do with this.

@ShadowMario he is using extra keys maybe

no, extra keys actually has this (fixed even), for some reason after the latest update, this issue just fixed itself, so I'm closing this


Group endpoint spec

This should fix the swagger documentation for the PUT /user/<int:user_id>/groups endpoint.

Perfect - thanks Ivan.


Laopa word list

https://zenodo.org/record/1123328#.X3OkXpUpBXg

There's more similar datasets from the uploader Andrew Hsiu on zenodo, for different languages


Could there be a way to add custom limits?

Is your feature request related to a problem? Please describe.
I have no way of adding additional limits for extensions for pterodactyl
Describe the solution you'd like
Ability to add custom limits like serverBuilder.customLimits({splits: 3})
Adding just custom data along with the ptero fetch request you know ;)
Describe alternatives you've considered
Raw pterodactyl fetch requests stink, gotta make a whole other request for a simple thing
Additional context
I have an extension for splitting servers, would like to use it along this library

I've never used custom limits, so how do they work, how are they set?

@SuperRonanCraft do you have an example request by chance?


Bug in raising NotSequenceError

There seems to be a bug in sequence validation.
Here's the stack trace from validating a yaml file where a field was supposed to be a sequence of objects but was in fact an object:
Traceback (most recent call last):
  File "c:\python27\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Python27\Scripts\pykwalify.exe\__main__.py", line 9, in <module>
  File "c:\python27\lib\site-packages\pykwalify\cli.py", line 84, in cli_entrypoint
    run(parse_cli())
  File "c:\python27\lib\site-packages\pykwalify\cli.py", line 71, in run
    c.validate()
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 145, in validate
    errors = self._start_validate(self.source)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 190, in _start_validate
    self._validate(value, root_rule, path, errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 208, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 303, in _validate_sequence
    self._validate(item, r, "{}/{}".format(path, i), tmp_errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 208, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 303, in _validate_sequence
    self._validate(item, r, "{}/{}".format(path, i), tmp_errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 210, in _validate
    self._validate_mapping(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 524, in _validate_mapping
    self._validate(v, r, u"{}/{}".format(path, k), errors, done)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 208, in _validate
    self._validate_sequence(value, rule, path, errors, done=None)
  File "c:\python27\lib\site-packages\pykwalify\core.py", line 285, in _validate_sequence
    raise NotSequenceError(u"Value: {} is not of a sequence type".format(value.encode('unicode_escape')))
AttributeError: 'dict' object has no attribute 'encode'

which points to:
[https://github.com/Grokzen/pykwalify/blob/unstable/pykwalify/core.py#L285]

If possible, can you provide a schema.yaml and data.yaml that cause this bug? Without this, it is hard for me to track down the error.
I think i get why this happens and what to do, but is possible i would need that.

Hi @pierre-borckmans
I do think that the following commit fixes this issue, https://github.com/Grokzen/pykwalify/commit/63fc9c9786e0ee05adf50c781d0b5aa7ce402d62 the updated behaviour only tries to encode string objects and not all object types.


data handling update

topTaxa
getAbundanceSample
getAbundanceFeature

For getTopTaxa the taxonomic ranks it is challenging because of the high number of missing information in the hierarchical rowData, especially as we go to lower taxonomic ranks. It may not be the ideal output especially if we use unique(). I think we can address this in tutorials.
E.g.
library(mia)
library(dplyr)
data(GlobalPatterns, package = "MicrobiomeExperiment")
me <- GlobalPatterns
top_features <- getTopTaxa(me, method="median", top=10)
tax_data <- rowData(me)[top_features,taxonomyRanks(me)]
tax_data %>% as.data.frame()

The format_to_besthit() is a utility to change rownames in both otu_table and tax_table to include taxonomic identity.
In the example below, I put getTaxonomyLabels to use in a different way, as it does not take a list of Ids as input to return their best taxonomy. Fetching of best taxonomy from a list of IDs can be a useful utility.
E.g.
library(mia)
library(pheatmap)
data(GlobalPatterns, package = "MicrobiomeExperiment")
me <- GlobalPatterns

## Example heatmap-1
pheatmap(assay(me, "counts")[26:50,1:20])
# Here, we see only IDs. Interpretation is limited with taxonomic information.

# Example heatmap-2
# Some processing
feature_tab <- assay(me, "counts")
# change IDs to best available taxonomic classification
rownames(feature_tab) <- getTaxonomyLabels(me)
feature_tab[1:6,1:6]
pheatmap(feature_tab[26:50,1:20])
# Here, we see taxonomy and higher interpretation is possible.

# Example heatmap-3
feature_tab2 <- assay(me, "counts")
rownames(feature_tab2) <- paste0("ASV:",rownames(feature_tab2), "-", getTaxonomyLabels(me))
pheatmap(feature_tab2[26:50,1:20])
# Here, we see IDs + taxonomy making this information richer and interpretation can be done specific to IDs and taxonomy.

If possible, How can we store this ? or Change rownames in rowData and assays to use the best taxonomic classification instead of some less-informative IDs e.g. "549656","331820" or "ASV001", "ASV002" ?
Let me know what you think of this.
Best wishes,
Sudarshan

I think you are looking for this:
# this changes rownames of assays as well as rowData(). Basically the equivalent to format_to_besthit()
rownames(me) <- getTaxonomyLabels(me)
feature_tab <- assay(me, "counts")
feature_tab[1:6,1:6]
pheatmap(feature_tab[26:50,1:20])

Right! :smile: getTaxonomyLabels works fine. Small edits/tweaks can be shown in tutorials.

Hi Sudarshan @microsud,
I cleaned up and simplified the code a bit. Please have a look the changes and let me now, if you agree with the changes. I will merge this once I get the confirmation.
If you have specific questions about certain changes, let me know as well. Happy to discuss this further.
Felix

These changes are okay with me!

Please have a look at the changes. Next time, the indentation must be OK, as well as other formating stuff. Therefore, have a look at changes for reference.

yes. I checked it. I had used formatR::tidy_app for formatting the code with 4 spaces. Next time will manually format as per your specification. For the @  in Roxygen labels will also refer to this code.


Move linter and build configs to scripts folder

This tidies up our root folder as much as possible (other files can't be moved as far as I'm aware).

I think the only thing we want here is moving the Webpack config into the client/ dir.

We sure can do that, but will be easier to make a new PR later.


Warning: [react-router] Location "/repo-name/" did not match any routes

Thanks a lot for this project!
now if I could only get it to work ;)
I have a react SPA hosted on gh pages, and I have two repos for it:

"original": Used for dev (repo) and hosting the production app on gh-pages
"staging": used only for hosting the staging app on gh-pages branch

I've bundled and deployed this commit from repo 1 to staging repo gh-branch.
tho when I go to https://goldylucks.github.io/memory-n-back-staging/ I get in the console Warning: [react-router] Location "/memory-n-back-staging/" did not match any routes
I've copied the [404.html](https://github.com/goldylucks/memory-n-yoru sback-staging/blob/gh-pages/404.html) like u said, and tried setting pathPrefix to both false/true
Also I copy pasted the script to the index.js (before the call to the spa bundle)
If it matters for any reason, I also have react-native and a nodejs a server code on that repo, tho to gh-pages only the react app and relevant assets are deployed
That's all the information I can think of that is relevant
any ideas what I'm missing?
Thanks again for this project and sry if I'm missing something really dumb :hankey:

This is an issue with how you set up your routes. React Router works based on the full absolute path, so you need to add the repo-name prefix to your routes and links, for example:
<Route path="/memory-n-back-staging/about" component={About} />
<Link to="/memory-n-back-staging/about">About</Link>
Also, you should set pathPrefix to true for the redirect to work.

gotcha, sry for wrongfully opening the issue
did u ever get it to work with non custom domain gh-pages ?

btw I'm trying this with the same website just hosted on surge and it also doesn't work. u c any reason it won't work on surge?

Yes it works fine with non custom domains for gh-pages, you just have to prefix Routes and Links with the repo name and set pathPrefix to true.

Yes it works fine with non custom domains for gh-pages, you just have to prefix Routes and Links with the repo name and set pathPrefix to true.

r u monkey-patching the <Route> and <Link> for this?
or passing a constant to all of them i.e.
// routes.js
import { GH_PAGES_PREFIX } from 'path/to/constants
<Route path={ `${GH_PAGES_PREFIX}/mypath` } component={ MyComponent }>

// constants.js
export const GH_PAGES_PREFIX = isProd ? 'repo-name' : ''

Not monkey patching. Just using a repo-name constant.

You can also use the basename option when creating your history, so you don't have to do this.


Typo Fix

I got lazy sorry,

Please do follow the pull request description given to you as it makes it easier for us to review

Looks like the whole file will need updating to our guidelines, as the CI is complaining about it. Would you like to work on this, @mmvergara, or create an issue for the same?

i'd like to work on it but sorry i don't really know how CI works, im kinda new to github. i might ruin something.

Looks like the whole file will need updating to our guidelines, as the CI is complaining about it. Would you like to work on this, @mmvergara, or create an issue for the same?

i'd like to work on it but sorry i don't really know how CI works, im kinda new to github. i might ruin something.

no no he is asking you to fix the whole file and make it look better.
ie documenting code

What  did i just do lol, sorry i really don't know how the flow in these PR's work 💀

you merged your pr with the main branch! i.e update your pull request so its up to date

Please use climits instead of limits.h.
Do not use using namespace std. Use using <function> (such as std::cout) or directly their namespace name.
Make sure to provide documentation and explanation of the algorithm, author, links, and self-test implementations.

If you need any help with this, let us know! 🙂

aight yeah. can someone actually finish this PR for me, sorry i'm not so used to github or C++ or this algorithm (yet).

I'll work on this in the next few days. Thank you for contributing. 🙂


SAK-42248 Fix empty error banner displaying when uploading/importing files

Added display:none to prevent empty error banner from displaying while uploading files, similar to https://jira.sakaiproject.org/browse/SAK-41516 (https://github.com/sakaiproject/sakai/pull/6706).
Should fix bugs https://jira.sakaiproject.org/browse/SAK-42248 and https://jira.sakaiproject.org/browse/SAK-42121.

Leaving the inline style="display:none" and modifying the velocity template to look for sak-banner-error should preserve the intended functionality of the original alertMessage & prevent the empty error banner from displaying. Just updated the PR to fix the copyright function.
@ern  @fostersdesign @profmikegreene


some question

May I ask if there is more detailed guidance here? I don't know how to operate this project!

can you give me the file about Train_GCC-training_output.csv and
Validation_GCC-1.1.0-Validation_output.csv! Thank you!


Support Arm Mac

Doxygen started recently releasing native Apple Silicon builds: https://github.com/doxygen/doxygen/issues/10066#issuecomment-2483706108
Supporting this would solve the CI/CQ tryjob issues I described in https://github.com/TendTo/rules_doxygen/pull/6#issuecomment-2479509517

My first instinct is to split the mac platform into mac_arm and mac_intel:
doxygen_extension.version(
    sha256 = "...",
    # url needs to point to github action artifact download link for now
    url = "...",
    platform = "mac_arm"
)

doxygen_extension.version(
    sha256 = "...",
    version = "1.9.6",
    platform = "mac_intel"
)

@TendTo also suggested exposing executable here https://github.com/doxygen/doxygen/issues/10066#issuecomment-2483832777
I think either would work for us but splitting the platforms into mac_arm and mac_intel would be a bit easier for me at the moment

@TendTo re: your idea to expose executable, would this allow me to specify a different executable platform? I'm pretty sure that would work for us.
Re: the rules_doxygen API it seems like the proper way to expose these permutations might be:
doxygen_extension.version(
    executable = "...",
    os = "mac",
    arch = "aarch64",
)

I left a comment in https://github.com/doxygen/doxygen/issues/10066 asking if they plan on releasing Doxygen 1.13.0 soon (and including the macOS ARM binary as part of the release). If they are, then it may be best to just wait for that. This issue is turning out to be complicated and I'm still figuring out what's the best course of action for Pigweed here.

Doxygen confirmed that they don't have plans any time soon to provide official macOS ARM binary releases so I think that settles the issue. The path forward IMO is for rules_doxygen to allow users to provide their own per-platform executables
https://github.com/doxygen/doxygen/issues/10066#issuecomment-2494658631

Ok, I think we may be close. There are 2 new configurable platforms, linux-arm and mac-arm, and you can now provide an executable in the configuration. It should be a Label pointing to it, and you may need a @@ at the beginning if you are referring to the root module.
I also took advantage of this and also introduced a very easy to fix breaking change, (just rename doxygen_extension.version -> doxygen_extension.configuration).
Let me know if this would work for you.

Great, thank you! I'll provide an update later today.

I'm not sure if whether I'm holding something wrong or it's something on the rules_doxygen end but I'll share my progress.
In MODULE.bazel I've rigged up the prototype workflow to download/extract binaries:
http_archive(
    name = "doxygen_linux_gcc",
    url = "https://kaycebasques.github.io/doxybin/20241125/ubuntu-gcc-release.zip",
    sha256 = "f479c0eed0b0de28f58c6c7997c38d282b7afadf73dd132199019e4ff7f100f2",
    type = "zip",
    build_file_content = """
filegroup(
    name = "files",
    srcs = ["doxygen"],
    visibility = ["//visibility:public"],
)
sh_binary(
    name = "executable",
    srcs = ["doxygen"],
    data = [":files"],
    visibility = ["//visibility:public"],
)
""",
)

bazelisk run @doxygen_linux_gcc//:executable works as expected
also in MODULE.bazel I've patched it to use latest code like so:
git_override(
    module_name = "rules_doxygen",
    commit = "b8758dd1f8d54b6024e05c181b0a6f35d57e91e6",  # presumably I just use last commit from the WIP PR?
    remote = "https://github.com/tendto/rules_doxygen.git",
)
doxygen_extension = use_extension("@rules_doxygen//:extensions.bzl", "doxygen_extension")
doxygen_extension.configuration(
    platform = "linux",
    executable = "@doxygen_linux_gcc//:executable",
)

Trying to build the docs with the self-provided binary however leads to this error:
ERROR: Traceback (most recent call last):
	File "/home/kayce/.cache/bazel/_bazel_kayce/9659373b1552c281136de1c8eeb3080d/external/rules_doxygen+/extensions.bzl", line 258, column 44, in _doxygen_extension_impl
		executables.append(str(ctx.path(Label(attr.executable))) if attr.executable != "" else "")
Error in path: Unable to load package for @@[unknown repo 'doxygen_linux_gcc' requested from @@rules_doxygen+]//:executable: The repository '@@[unknown repo 'doxygen_linux_gcc' requested from @@rules_doxygen+]' could not be resolved: No repository visible as '@doxygen_linux_gcc' from repository '@@rules_doxygen+'
ERROR: error evaluating module extension doxygen_extension in @@rules_doxygen+//:extensions.bzl
INFO: Elapsed time: 0.112s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
FAILED:
    Fetching module extension doxygen_extension in @@rules_doxygen+//:extensions.bzl; starting

Unfortunately I don't really understand all the details behind repository visibility at play here.
Based on my experimentation, try adding another @ before the label for the executable. It would become @@doxygen_linux_gcc//:executable.


Move filter_push_down::split_members to be reused outside of DataFusion

The filter_push_down::split_members function is being copied and used outside in IOx. If this function is moved to optimizer/utils, it will be reused outside DataFusion

Maybe it could be a function on Expr itself

This issue seems obsolete, given that no split_members function exists anymore, so IoX must have worked around that.

Closing this. Thanks @drauschenbach.


call-compartments: object 'weight' doesn't exist

Hi, I run "cooltools compute-expected" successfully with "--weight-name KR". Then I realized "cooltools call-compartments" doesn't have this option. I got the error described in the title when running "cooltools call-compartments". Do you have any suggestions to solve this?
   +-- 500000          +-- bins          |   +-- GW_KR (6189,) float64          |   +-- GW_SCALE (6189,) float64          |   +-- GW_VC (6189,) float64          |   +-- INTER_KR (6189,) float64          |   +-- INTER_SCALE (6189,) float64          |   +-- INTER_VC (6189,) float64          |   +-- KR (6189,) float64          |   +-- SCALE (6189,) float64          |   +-- VC (6189,) float64          |   +-- VC_SQRT (6189,) float64          |   +-- chrom (6189,) int32          |   +-- end (6189,) int32          |   +-- start (6189,) int32          +-- chroms          |   +-- length (25,) int32          |   +-- name (25,) |S32          +-- indexes          |   +-- bin1_offset (6190,) int64          |   +-- chrom_offset (26,) int64          +-- pixels              +-- bin1_id (16288151,) int64              +-- bin2_id (16288151,) int64              +-- count (16288151,) int32

Hi! The option to choose the weight column actually exists, but isn't exposed via the CLI yet.
Until it's available, you can try replicating what the CLI is doing from Python, adding balance="KR" to the function call: https://github.com/mirnylab/cooltools/blob/master/cooltools/cli/call_compartments.py#L148-L166
If by any chance you'd be interested in making a PR, let me know. The solution would be to expose these same two options to call-compartments. Note that balance=True defaults to the column name "weight", and balance={some string} uses a custom weight column name.

Great! Thanks for the help.


test: fixing another integration flake on macos

the disconnect due to match can occur before the write happens.  don't validate write success.
Risk Level: n/a (test only)
Testing: yes
Docs Changes: no
Release Notes: no

coverage failure was unrelated.  sent out https://github.com/envoyproxy/envoy/pull/11961 to try to fix

/azp run


Lab 15 Observations

Module: 7 – Integrate Azure Cosmos DB for NoSQL with Azure services
Lab/Demo: 15 – Search data using Azure Cognitive Search and Azure Cosmos DB for NoSQL
Task: Create Azure Cognitive Search resource
Step: 1
Azure Cognitive Search has been renamed to Azure AI Search.
Looking for cognitive search gives Azure Cognitive Search solution for Sentinel as the first result. This is (obviously) not the intended resource.
Only the second result (Azure AI Search) mentions on the card … “formerly Azure Cognitive Search”.
Task: Validate index with example search queries
Generally: as the search field/method has changed, lots of the steps do no longer work.
When I ran first through the labs, I did some digging into the JSON(?) syntax and could make most or even all of the steps work.
Step: 5
No count field.
Step: 7
top is ignored.
Step: 9
Pagination not working.
Step: 11
Price filter not working.
Step: 13
Buckets not working.

@pstueck, thank you for bringing this to our attention.  Lab has been fixed to reflect the correct name of the product, and to use the JSON view to return the correct data.


Introduce a new Calculation Mode pick-list to reflect "Schedule Calculate" scenarios

Users that want to use this facility, get confused about which Calculation Mode to pick, typically the Process Builder choice is the best, but this is far from obvious. More confusing is that the picklist already has a Scheduled entry in it (which is quite different from Schedule Calculate). Maybe have "Scheduled Full" ?

Yes I agree! I'm confused right now about the Calculation Mode. I have created two Rollups that need once a week recalculation. I scheduled them, but what mode do I need to select now? I thought PB was a good one, but started doubting because it's actually a scheduled Rollup but they should NOT be included in the general Scheduled Rolups.

If you used the 'Schedule Calculate' button for a full recalc each schedule, just set Calculation Mode to PB and you don't need to deploy a trigger either.
If you want scheduled recalc only of parent records the tool determines need to be recalculated, set Calculation Mode to Scheduled and use the Apex Classes page to scheduled the general rollup scheduler. This will use records stored in the scheduled line items object to determine what needs to be recalculated.
The later option is lighter on the system and will run faster each schedule. However like realtime, the logic to detect whats changed and thus what parents need recalc is flawed by time based formula functions in your criteria. Thus full recalc in these situations is better.
Hope this helps.


Add new attribute to existing block

Hi, how can I add new attribute to existing block in model space?

First we need to differentiate between a block reference (aka INSERT, that is how it is called in the DXF) and a block definition (aka BLOCK). So, I guess when you say block what you really mean is Insert.
To add an Attribute to an Insert first you need to add its corresponding AttributeDefinition to the Block associated with the Insert and then synchronize it. As shown in this sample:
// pick the block definition from the document, or get it directly from the Block property of the Insert
Block block = doc.Blocks["MyBlock"];

// create the new attribute definition
AttributeDefinition attDef = new AttributeDefinition("AttDef02")
{
    Position = new Vector3(50, 50, 0),
    Prompt = "Insert value:;",
    Value = 0,
    Height = 5.0
};

// add it to the block attribute definitions
block.AttributeDefinitions.Add(attDef);

// pick the insert you want or iterate through the collection if you want to edit all of them
Insert insert = doc.Inserts.ElementAt(0);
// synchronize the insert will reset all insert attributes to their default values
insert.Sync();
// after synchronizing all insert attributes will be reseted to the default values
// you will need to manually reassign any modified property
insert.Attributes.AttributeWithTag("AttDef02").Value = 10;


Rewrite Order Status Functionality

Create a more thought-out order status system that is flexible for developers to implement their own custom order pipelines.

Possibly have a 'financial status', as well as other status like 'fulfillment status'.

Could also introduce the concept of a line being fulfilled (shipped)...or status per line.

hello, I have some questions, for example, order system is very common, but how do we store the statemmache status for every order when we use the statemachine in our order system?


Error: missing "=" in connection info string

I'm connecting to postgres via a unix file socket in $PGHOST/.s.PGSQL.5432 which is ~/tmp/.s.PGSQL.5432
diesel migration run and infer_schema!("dotenv:DATABASE_URL") worked.
But when running my application, I get:
2017-01-27T12:23:50.729025729+01:00 ERROR r2d2 - missing "=" after "postgres://user:pass@/dbname" in connection info string

2017-01-27T12:23:50.729091443+01:00 ERROR r2d2 - missing "=" after "postgres://user:pass@/dbname" in connection info string

thread '<unnamed>' panicked at 'Failed to create pool: InitializationError(Some("missing \"=\" after \"postgres://user:pass@/dbname\" in connection info string\n"))', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libcore/result.rs:870
note: Run with `RUST_BACKTRACE=1` for a backtrace.
thread '<unnamed>' panicked at 'Once instance has previously been poisoned', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/sync/once.rs:263
thread '<unnamed>' panicked at 'Once instance has previously been poisoned', /buildslave/rust-buildbot/slave/nightly-dist-rustc-linux/build/src/libstd/sync/once.rs:263

I found out I had to use postgres://user:pass@%2Fhome%2Fnosgo%2Ftmp/dbname


Not App Container Specification compliant

My client requires we use a container image system that will allow them to change the back-end at their choosing... ie they want to avoid "vendor lock-in". Docker is the only thing that is compatible with Docker... making it a no-go. Please update Docker to be compatible with ACI so that we can continue to use your product.
https://github.com/appc/spec/blob/master/SPEC.md

This has been discussed in the past: #9538 #10131

Hi @davetinder. I'm afraid your client was scammed. Every popular open-source project goes through a phase where its competitors feel threatened, and try to convince the community to stop using it. Coming up with a new "standard", then criticizing the established open-source project for failing to implement it, is a common tactic. Of course, the problem is that 1) the so-called standard is immature, unproven, and has virtually zero actual adoption; and 2) there is no actual risk of vendor lock-in, since the project is entirely open-source and not controlled by a single vendor. Docker has several contributors employed by Red Hat, Google, Microsoft, IBM and many small and mid-sized companies. If Docker, inc. went out of business or changed the format tomorrow, the project would live on.
The same is true for Linux, Git and most other popular open-source projects. Sure, Linux could decide to break every single API in the next release of Linux. But within the week, most of the Linux community would probably flock to a new project designed to maintain reverse compatibility. Would you call that vendor lock-in? Docker has received contributions from nearly 1,000 people. Each release (we release every 2 months) involves nearly 100 people. And we keep growing every day.
The Docker project encourages alternative implementations - both of its runtime environment and content distribution formats. We already publish specifications for both (see the project documentation). Could those specifications be improved? Yes. But there are already several alternative implementations in use at large scale, proving that vendor lock-in with Docker is a myth.
One last fact, which you might find funny: one of these alternative implementations of Docker's image distribution system is developed by CoreOS, the very same vendor which is propping up this so-called standard (for the competitive reasons I explained above). They use it to power a competitor to Docker Hub (the commercial offering of Docker, inc.). So, the very people trying to convince you that it's impossible to implement the Docker standard without being a customer of Docker, inc. are doing exactly that to compete with Docker, inc. And even though they are a competitor, the Docker community still actively supports them.

Note to other maintainers. There is a high probability that this is a spam account. We just noticed a pattern of new github accounts, created for the sole purpose of requesting support for the "app container specification", then never used again.
On the one hand I want to give the sponsors of this "standard" the benefit of the doubt. Maybe those were genuinely new github users, which decided to create an account specifically for this, and then never had another use for their account. It is entirely possible.
But just in case: please keep an eye out for bogus spam account creating the illusion of demand for this vendor-backed "standard" called "application container specification".

See also https://github.com/Jeff85


User Settings: files.encoding not working if opened from contextmenu

VSCode Version: 1.43.2 (user setup)
OS Version: Windows 10

Steps to Reproduce:

configure language specific option for objectpascal
set "files.encoding" : "windows1252"
Open .pas-file from the VSCode File-Open Menu - it opens with windows1252 encoding  as expected.
Open .pas-file from the File-Explorer ContextMenu - it opens with utf-8 encoding but I set it to open with windows1252!

Does this issue occur when all extensions are disabled?: Yes

Open .pas-file from the File-Explorer ContextMenu - it opens with utf-8 encoding WRONG

How exactly?

@bpasero - please see my edit

Open .pas-file from the File-Explorer ContextMenu - it opens with utf-8 encoding but I set it to open with windows1252!

I was asking, what context menu?

I don't understand your question?!?
Explorer.exe (aka FileExplorer or FileManager), go to any xyz.pas file, right click and than "Open with Code".

Understood. This is likely an issue with timing: when you open the file into a new VSCode instance, at the time the file opens, we are not aware of the object pascal extension because it only loads and activates later. But at that point the file is already opened with the default encoding.
Does it help if you configure "window.openFilesInNewWindow": "off" so that the file opens into an existing instance? Does that even work to open into an existing instance?

This works!
Can I setup an encoding, not for ObjectPascal but for *.pas - files? Than the Extension must not be loaded to aware the right encoding.

@Andyriod yeah we have that as open request in https://github.com/microsoft/vscode/issues/35350 and I suggest we close this one as duplicate, I do not have a better suggestion right now.


Cannot use Azure OpenAI embedding model.

I cannot use Azure OpenAI embedding model as:
const azureOpenaiLLM = new OpenAI({ model: "gpt-4", temperature: 0 });
const azureOpenaiEmbedding = new OpenAIEmbedding({ model: "text-embedding-ada-002", });
const serviceContext = serviceContextFromDefaults({ llm: azureOpenaiLLM, embedModel : azureOpenaiEmbedding });
looks like the value not pass into:
Stack trace:
Error: 400 The embeddings operation does not work with the specified model, gpt-35-turbo. Please choose different model and try again.
May I know how to use Azure OpenAI embedding model?

I assume you set the Azure env variables, see https://ts.llamaindex.ai/modules/llms/available_llms/azure#environment-variables?
Then I would try calling the embedding model directly first:
  const texts = ["hello", "world"];
  const embeddings = await azureOpenaiEmbedding.getTextEmbeddingsBatch(texts);
  console.log(`\nWe have ${embeddings.length} embeddings`);

@xuhaoruins when creating an instance of OpenAIEmbedding class, you need to make sure you are using EMBEDDING_MODEL as a deployment target:
  const azure = {
    azureADTokenProvider,
    deployment: process.env.AZURE_OPENAI_DEPLOYMENT ?? "gpt-35-turbo",
  };

  // configure LLM model
  Settings.llm = new OpenAI({
    azure,
  }) as any;

  azure.deployment = process.env.EMBEDDING_MODEL as string; // <-----

  Settings.embedModel = new OpenAIEmbedding({
    azure,
    model: process.env.EMBEDDING_MODEL,
    dimensions: process.env.EMBEDDING_DIM
      ? parseInt(process.env.EMBEDDING_DIM)
      : undefined,
  });

@xuhaoruins does this mean that azure.deployment and model need to have the same value? If it's not possible to use e.g. my-embedding as deployment name then we would need to change that.


Add System.register as a format target

For production builds System.register is a better option long-term.  I keep running into weird cases with having a bunch of named AMD defines in a single file, I don't feel confident that it's always going to be viable to go that way.

@matthewp I would prefer to write to AMD because its a format that's supported by other loaders.  It's not "owned" by SystemJS and designed for "jspm".
But, for globals, this is something we might need to do sooner than later.  I was able to get a global to work by writing out something like:
System.register("slave", [], false, function(){
	var load = {source: "window.slave = {name: 'slave'};", name: "slave", metadata: {format: "global", exports: "slave", deps: []}};
	var exportName = load.metadata.exports;

	var loader = System;

	loader.get('@@global-helpers').prepareGlobal(load.name, load.metadata.deps);

    if (exportName)
      load.source += '\nthis["' + exportName + '"] = ' + exportName + ';';

    // disable AMD detection
    var define = loader.global.define;
    loader.global.define = undefined;

    // ensure no NodeJS environment detection
    loader.global.module = undefined;
    loader.global.exports = undefined;

    loader.__exec(load);

    loader.global.define = define;

    return loader.get('@@global-helpers').retrieveGlobal(load.name, exportName, load.metadata.init);
});

instead of:
System.define("slave","window.slave = {name: 'slave'};",{"metadata": {"format": "global"}});

I need this for an issue I'm having, so I'm going to implement this.

I got this to work for globals using the same technique in systemjs/builder. I see no reason why it has to be System.register though, this will work as a define as well. So I'm going to try this next.


brew cleanup doesn't remove incomplete downloads

Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.

[x] are reporting a bug others will be able to reproduce and not asking a question. If you're not sure or want to ask a question do so on our Discourse: https://discourse.brew.sh
[x] ran a brew command and reproduced the problem with multiple formulae? If it's a problem with a single, official formula (not cask) please file this issue at Homebrew/homebrew-core: https://github.com/Homebrew/homebrew-core/issues/new/choose. If it's a brew cask problem please file this issue at https://github.com/Homebrew/homebrew-cask/issues/new/choose. If it's a tap (e.g. Homebrew/homebrew-php) problem please file this issue at the tap.
[x] ran brew update and can still reproduce the problem?
[x] ran brew doctor, fixed all issues and can still reproduce the problem?
[ ] ran brew config and brew doctor and included their output with your issue?

HOMEBREW_VERSION: 1.7.6-105-gff3cede
ORIGIN: https://github.com/Homebrew/brew
HEAD: ff3cede96f6930bb3a0e4711c47724bb2b3d4bc1
Last commit: 7 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 63966af9c41a16540cfedad660689ef0420cf3a4
Core tap last commit: 3 hours ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_GIT: /usr/local/opt/git/bin/git
HOMEBREW_BINTRAY_KEY: set
HOMEBREW_BINTRAY_USER: commitay
HOMEBREW_CASK_OPTS: --no-quarantine
HOMEBREW_DEVELOPER: 1
HOMEBREW_GITHUB_API_TOKEN: set
HOMEBREW_GIT_PATH: /usr/local/opt/git/bin/git
HOMEBREW_NO_ANALYTICS: 1
HOMEBREW_NO_AUTO_UPDATE: 1
HOMEBREW_NO_INSECURE_REDIRECT: 1
HOMEBREW_PRY: 1
HOMEBREW_VISUAL: subl -w
CPU: quad-core 64-bit sandybridge
Homebrew Ruby: 2.3.7 => /System/Library/Frameworks/Ruby.framework/Versions/2.3/usr/bin/ruby
Clang: 10.0 build 1000
Git: 2.19.0 => /usr/local/opt/git/bin/git
Curl: 7.54.0 => /usr/bin/curl
Java: 11, 1.8.0_181
macOS: 10.13.6-x86_64
CLT: <IP_ADDRESS>.1.1535735448
Xcode: 10.0
XQuartz: 2.7.11 => /opt/X11

$ lst $(brew --cache)
/Users/commitay/Library/Caches/Homebrew
├── Cask
└── downloads
$ brew fetch fd exa gcc
Fetching: fd, exa, gcc
==> Downloading https://homebrew.bintray.com/bottles/fd-7.1.0.high_sierra.bottle.tar.gz
######################################################################## 100.0%
Downloaded to: /Users/commitay/Library/Caches/Homebrew/downloads/6a80f21e6883e7347578b49b66fa06a03fa6810d2b8b0700307b5fe645730640--fd-7.1.0.high_sierra.bottle.tar.gz
SHA256: be21df7d1288190659327d2dc4329989fa5700c15a50aef6049b7b54d6ac0602
==> Downloading https://homebrew.bintray.com/bottles/exa-0.8.0.high_sierra.bottle.tar.gz
######################################################################## 100.0%
Downloaded to: /Users/commitay/Library/Caches/Homebrew/downloads/e8b63574ca3510b02cd815cbd076d7d6322723dd32831274e7324a5dfa8fd291--exa-0.8.0.high_sierra.bottle.tar.gz
SHA256: e10e5f5d86fbab6f6d558c620a73f1caf157d42c5a01c54636f2de08f4aaee4b
==> Downloading https://homebrew.bintray.com/bottles/gcc-8.2.0.high_sierra.bottle.1.tar.g
#####                                                                      7.9%^C
$ lst $(brew --cache)
/Users/commitay/Library/Caches/Homebrew
├── Cask
├── downloads
│  ├── 6a80f21e6883e7347578b49b66fa06a03fa6810d2b8b0700307b5fe645730640--fd-7.1.0.high_sierra.bottle.tar.gz
│  ├── cee243f87b927a91df9f84cacf3e56ed7d0dff977a8fc34d42aef6d22b90312b--gcc-8.2.0.high_sierra.bottle.1.tar.gz.incomplete
│  └── e8b63574ca3510b02cd815cbd076d7d6322723dd32831274e7324a5dfa8fd291--exa-0.8.0.high_sierra.bottle.tar.gz
├── exa--0.8.0.high_sierra.bottle.tar.gz -> downloads/e8b63574ca3510b02cd815cbd076d7d6322723dd32831274e7324a5dfa8fd291--exa-0.8.0.high_sierra.bottle.tar.gz
└── fd--7.1.0.high_sierra.bottle.tar.gz -> downloads/6a80f21e6883e7347578b49b66fa06a03fa6810d2b8b0700307b5fe645730640--fd-7.1.0.high_sierra.bottle.tar.gz
$ brew cleanup --prune=0
Removing: /Users/commitay/Library/Caches/Homebrew/exa--0.8.0.high_sierra.bottle.tar.gz... (635.8KB)
Removing: /Users/commitay/Library/Caches/Homebrew/fd--7.1.0.high_sierra.bottle.tar.gz... (1MB)
==> This operation has freed approximately 1.7MB of disk space.
$ lst $(brew --cache)
/Users/commitay/Library/Caches/Homebrew
├── Cask
└── downloads
   └── cee243f87b927a91df9f84cacf3e56ed7d0dff977a8fc34d42aef6d22b90312b--gcc-8.2.0.high_sierra.bottle.1.tar.gz.incomplete

@commitay For the sake of tracking mind including your brew config? If you could git bisect this that'd be handy too.

Added brew config. The incomplete file in HOMEBREW_CACHE/downloads is never removed when bisecting so I think this problem was introduced with the cache changes.


[Bug] No Syntax highlighter for markdown in docs?

Insomnia Version: 5.16.6
Operating System: Win10

Details
Writing a code block in markdown with specific language (like ```java), there seems to be some form of syntax highlighting in "WRITE" tab but everything is gone in "PREVIEW" or in actual "DOCS" tab.
Can this be fixed?
Thanks.

Hmm, this should be possible with something like highlight.js.

Actually, highlight is already in there. Maybe there's just no support for Java.

Hmm, nope. Java seems to be working for me

It looks not very many of the token types have associated colors. I will add some more

There, that's a bit better


Import version.version in init.py

Codecov Report

Merging #55 (1a7d229) into main (b67a60a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main      #55   +/-   ##
=======================================
  Coverage   99.48%   99.48%
=======================================
  Files          18       19    +1
  Lines         585      587    +2
=======================================
+ Hits          582      584    +2
  Misses          3        3

Flag
Coverage Δ

unittests
99.48% <100.00%> (+<0.01%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

runtime/databricks/automl_runtime/__init__.py
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b67a60a...1a7d229. Read the comment docs.


Unable to upload files !

Droppy Version 5.1.9 shows Unable to upload when any file is added for uploading.

Few questions:

Which browser are you using?
Are you uploading through the button or drag-and-drop?
Are you uploading a single file, a directory, or a mix of both?
Does https://silverwind.io/uppie/example.html output the file paths (no files are uploaded in that demo)?

Answers to the questions:

I am using Mozilla Firefox.
I am uploading through the "Add files" buttons.
I am uploading a single file.
Yes. https://silverwind.io/uppie/example.html output the file paths (no files are uploaded in that demo) as show in the attached screenshot:

The correct behaviour on the demo should be that the filename(s) are logged below the input element. I've expanded https://silverwind.io/uppie/example.html to support file and directories, can you retry with both buttons and also watch for any errors in the DevTools console?

I have retried with both buttons on your mentioned link. Here are the observations:

Mozilla Firefox shows unexpected behaviour and shows "TypeError: files is undefined" in Developer Web Console. Directories button can't add directory. The Directories button acts same as Files button and shows the same "TypeError: files is undefined" in Developer Web Console.

Google Chrome shows correct behaviour as you mentioned for both buttons.
For clarification I have attached the screen-shots:

Mozilla Firefox Single File upload:

Mozilla Firefox TypeError: files is undefined:

Mozilla Firefox Directories upload:

Google Chrome Single File upload:

Google Chrome Directory upload:

Thanks, I'll investigate further. Could be that I missed something incompatible on the Release line of Firefox. The directory button will work in the next Firefox release (50).

Can reproduce now, it looks like the pref dom.input.dirpicker which I had enabled was for some reason interfering with the file picker.

Should I try again to reproduce those in your given link?
Or should I try my local installation of Droppy to check again?

No action needed, I'm building v5.1.11 now which will resolve this bug.

And v5.1.11 is out, please update!


Ability to retrieve GraphQL arguments from WebGraphQlInterceptor

I'm trying to implement a WebGraphQlInterceptor for authorizing requests based on a GraphQL argument. The argument is dynamic and may vary for a given user, so passing it via a header property doesn't make sense.
If the argument is passed in the variables section of the GraphQL request, I can access it just fine as below:
@Component
@Slf4j
public class MyAuthorizationInterceptor implements WebGraphQlInterceptor {
    @Override
    public Mono<WebGraphQlResponse> intercept(WebGraphQlRequest request, Chain chain) {
        var clientIdVariable = request.getVariables().get("clientId");
        // Authorize request
        // .....

        return chain.next(request);
    }
}

However, since I can't force my clients to use variables over inline GraphQL arguments, I need to be able to fallback and grab the clientId argument if it's not found in the variables. From what I've been playing with and finding online, I'm not sure this is supported very easily. Of course, I can use a GraphQL parser to convert the GraphQL document to a tree structure and manually get the client ID, but that's a lot of overhead that seems unnecessary -- I would have thought this would be supported out-of-the-box from the WebGraphQlRequest.
Am I missing something? Is there an easy way to get the arguments even if they're not passed in the variables section? Are there alternatives I'm overlooking?

WebGraphQlInterceptor is a point of interception in the HTTP layer. At this level, we have a JSON request body that contains the GraphQL document (i.e. the textual representation of the request) along with variables and extensions. The GraphQL document has not yet been parsed. Parsing it is the job of the GraphQL engine to which the transport delegates to process the request. In short, not overlooking anything. This is just not available at this level.
You could look into using an Instrumentation which provides various callbacks for hooking into the GraphQL Java engine, as parses and validates the document

@rstoyanchev  Thanks for the info and suggestion. After looking into the instrumentation docs, it does look like an instrumentation validator is what I should be using instead of a Spring interceptor for my use case.


Do have any component to edit the pdf file ?

Version

Development Environment

Current Behavior

// Write example code

Expected Behavior

@sathishvikkas Nope. PDF file is not editable in ImageEditor


Environments in settings GUI

Description

first version of way how to store environments in setting gui
this PR allows to set environemnts only via raw-json item type
with key "env_group_key" is possible to set "environment group" ("maya", "maya_2020", "nuke", etc.)

it is allowed to set only one item with specific "env_group_key"

current implementation kept previous environment values BUT
values of environment group is overriden if has equivalent in system configs

EXAMPLE: if schema will contain raw-json item with "env_group_key": "maya" and values of this schema will be saved, environments from ~/pype/settings/defauls/environments/maya.json won't be used anymore and can be removed

but the ones saved via settings are ignored.

Should be fixed. I didn't save environments to studio overrides...

Nope still nothing. It shouldn't be dependent on studio overrides though. Even in default, we want to save some envs into settings

Nope still nothing.

So, if you'll add for example this item to any schema:
{
    "key": "something_something",
    "label": "Maya env test",
    "type": "raw-json",
    "env_group_key": "maya"
}

open settings gui, save values and then load:
from pype.api import environments

test_env = environments()
print(test_env["maya"])

The output is not what was saved in setting gui?

Precisely. I tried with
{
            "key": "ftrack_env_test",
            "label": "Ftrack Env Test",
            "type": "dict",
            "children": [{
                "key": "ftrack_env",
                "label": "Ftrack Environment",
                "type": "raw-json",
                "env_group_key": "ftrack_test"
            }

and then in nuke printing
from pype.api import environments
print(environments()['ftrack_env'])

and it's not there. I've save the env before as default and tried as studio override as well. I can see it saved in the settings but it doesn't load back.

and it's not there. I've save the env before as default and tried as studio override as well. I can see it saved in the settings but it doesn't load back.

Oh that's probably my fault, key is not store, but env_group_key, in your case it should be:
from pype.api import environments
print(environments()['ftrack_test'])

If it's too bizzar I can change env_group_key to boolean and key will be used as "group key"...


round to 3 decimals

round($pixels/$context)  is rounding. I'd like to have a round with 2 or 3 decimals. I'm using this plugin on maven..
sass::script::number.precision on Ruby works it seems. Is there a way i can configure in maven?

This version is no longer maintained. Refer to more recent forks at https://github.com/Jasig/sass-maven-plugin/network for more up to date versions.


[MM-12264] do not load redirect_location if enablelinkpreview is disabled

Summary
Currently if enablelinkpreview is disabled, the redirect_location url is still called
Ticket Link
https://mattermost.atlassian.net/browse/MM-12264
https://github.com/mattermost/mattermost-server/issues/9457
https://github.com/mattermost/mattermost-server/issues/9437
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[x] Ran make check-style to check for style errors (required for all pull requests)
[x] Has server changes (please link)
[x] Has UI changes

server changes https://github.com/mattermost/mattermost-server/pull/9461

I think the purpose of that additional condition check is to do API call (getRedirectLocation) only if necessary (that is if link preview is enabled).  As a result, we are effectively using the server traffic.
I believe we should submit the same condition check to mattermost-mobile repo - https://github.com/mattermost/mattermost-mobile/blob/master/app/components/post_body_additional_content/post_body_additional_content.js#L105

@saturninoabril could you make some note about that mattermost-mobile fix to jira. I am too lazy to do that PR :)

@zetaab this needs a rebase, whenever you can, please :)

little bit difficult to rebase because I used master branch :)

Thanks @zetaab!


Remove the deprecation and update the latest code base for litmus-python repo

Is this a BUG REPORT or FEATURE REQUEST?
Currently jinja2 is having deprecation error for Environment lib
Need to add requirement for markupsafe

https://github.com/litmuschaos/litmus-python/pull/41


chore(tools): Configurable coverage reporting

Feature based on discussion for a bug fix. Code coverage reporting will be a configurable option allowing the user to choose coverage based on TypeScript or the Transpiled JavaScript.

[x] Fix remap-istambul error
[x] Add sourcemaps to build.test ( Fixes #567 )
[ ] Move coverage to a configurable gulp task
[x] Investigate command line output configuration / both
[x] Verify zero percent coverage ( Fixes #560 )

I will finish this out once #622 lands

@d3viant0ne can we close this one? We can add a wiki page or reference to istanbul on how to configure the coverage report.

Yeah, sorry


Created PR

added new links for windows 10 1803 and 1809.  I am not sure the adjustment while inserting words and format, so I could not see the preview exact site after merging.
I took much care while editing this article. This is the tight work, it's not easy  to me

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: aa078bf6-a22e-a1e5-4dbd-f7401520d133
Version Independent ID: 9df06f8b-426c-b444-4478-c6bde84f50db
Content: Get support for Windows security baselines - Windows security
Content Source: windows/security/threat-protection/windows-security-configuration-framework/get-support-for-security-baselines.md
Product: w10
Technology: windows
GitHub Login: @Dansimp
Microsoft Alias: dansimp

@illfated .  @e0i . Please review my PR #6287 .
i think reviewing of this PR is not easy, because it contains different formatted style.
I am sure  added hyperlink is  correct

Looks good to me, I am only able to suggest improvements to your issue ticket title text & PR title text.
Suggestion:
Security Compliance Toolkit: 2018 missing in the version matrix

@RAJU2529 Please do not create blank issues to draw attention to Pull Requests.


⚠ SceneTime has degraded performance

In 4756520, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 2729 ms

Resolved: SceneTime performance has improved in 97e27cc after 6 minutes.


fix: do not register libp2p handler twice for same protocol

Motivation
Closes https://github.com/ChainSafe/lodestar/issues/6319
Description
In https://github.com/ChainSafe/lodestar/pull/5325 we have split out some functionality to be able to register dial only protocols but duplicate check to not register libp2p handler twice for same protocol has been moved in a function which does not even call libp2p.handle.
This changes adds it back in the right place and makes sure we do not register libp2p handler twice for same protocol.

Error logs are no longer observed in sim tests (sim-test-multifork-logs.zip)

:tada: This PR is included in v1.16.0 :tada:


Add XCM version to each compatible node in document

This document contains compatible parachains implemented into SDK.
We should do following:

Add this list into our official docs
Add XCM versions that Parachains are currently running into that list ✅

List can be found here: https://github.com/paraspell/xcm-sdk/blob/main/docs/supportedNodes.md

Versions into list were added in commit https://github.com/paraspell/xcm-sdk/commit/3b4760e3101443c5e306b7283cc3d9b7bde785d2

List was added into our official docs in commit https://github.com/paraspell/docs/commit/d0e5632f2e5a5bf73cc5c65a31d16e237c8c3c5d


FR update for onboarding and provinces

Updated French copy for first screen of onboarding and for province selection

Closing - I cherry picked in this commit to https://github.com/cds-snc/covid-shield-mobile/pull/341


0006-Publication strategy and guidelines

I think we're overdue to have a conversation about this. I've added more details in the document in this PR: please read this first.
This issue is not intended to discuss any one specific code, but just to talk more broadly on what we can do to incentivize publications and maintain accountability (e.g. make sure publications are actually finished and published, since software papers are often one or two spots away from the top of the "to do" list; we've had a few sit in draft, or sit on arXiv).
I'm adding a public note that @shyuep, @janosh and myself have been planning a new pymatgen paper. We have heard loud and clear from the pymatgen developer community that this is something that is desired. More specifics to follow on this, but I would prefer not to discuss that here, and not to discuss any other specific paper in this issue. This issue is intended for overall strategy only.

Thanks for this! This reads very good already!
I would like to directly add an additional suggestion: in many EU countries, software publications such as zenodo now also count as publications. It very much helps if the release notes include detailed information on the contributors (similar to what is now done for every pymatgen release). Maybe, we could also add some best practices in this regard.

I would also add that ultimately, those who want a software paper should take charge and write it. Saying it is priority no. 100 is not really a valid excuse. If someone wants the recognition and attribution, they can make it happen rather than wait for other people to make it happen for them.

While I agree that someone has to put in the work, it is good to know for new and maybe even old developers that these options exist and that they can work towards a (new) software publication. Thus, writing it down makes asking for publications much easier.


Workaround for operators catalog downloader.

The previous way to get operators catalog pages wasn't retrieving
all the pages. It can be tested manually in the browser that there are
many more pages than the calculated ones.
As we cannot rely on the total value to calculate the number of
operator pages, the only option is reading until we found a page without any
operator.
Includes index file renaming and page_size increase to 500 from #192

This is not a fix, but a workaround for the bug in the catalog api.

Closing this PR as it will not be needed.


Added 35 icons

@xster Implemented changes as discussed and also keeping the fork alive this time.

Thanks for your patience and contribution. Will merge this PR when the build tree is green.


Rebrand "Mastercoin" to "Omni"

The foundation has requested that the SP1 and SP2 tokens are rebranded from Mastercoin to Omni (MSC => OMNI).
Only user facing adjustments have been made to keep the scope limited.  Comments, functions and variable names have not been modified.

Notes

The protocol does not support renaming of properties at this stage.
The argument has been made that since MSC & TMSC were not created by the protocol and are instead hardcoded that this limitation does not apply.
The final decision for this PR is probably for the board?

It would be great, if you could quickly update the API docs as well.

The foundation has requested ... The final decision for this PR is probably for the board?

This is a bit contradicting (at least for me), because It's not clear to me, whether this was approved or not, and if so, why would it need another decision then?

TOMNI ...

This almost looks like the name TOMMY. >_<"

This is a bit contradicting (at least for me), because It's not clear to me, whether this was approved or not, and if so, why would it need another decision then?

I'm not 100% on who get's to make that decision.  For some background, @patrickdugan was talking about an (already underway) rebranding effort with exchanges to rename the token.  I raised that our reference (Omni Core) didn't have this rebranding of the MSC token & also that I hadn't been asked to make these changes in OmniChest.info.
I thus said I would submit a PR and asked everyone to please come and make their comments re rebranding the token on the PR here.
As I understood it from the meeting the rebrand of the token is already in full swing with third parties, so that would indicate at least tacit approval, but to be honest I'm a little shaky on authority/who is actually on the 'board' these days :(

TOMNI

This almost looks like the name TOMMY. >_<

Hehe yeah it'll take some getting used to!

We approved the name change, glad to see it’s moving forward.  Thanks, guys!

The tests currently fail because of the following OmniJ test:

https://github.com/OmniLayer/OmniJ/blob/master/omnij-rpc/src/integ/groovy/foundation/omni/test/rpc/basic/ListPropertiesSpec.groovy#L28

This was forseeable. I'm going to visually check the submission once more later, but so far I have no objections.

Hey guys, I take it we just need the repo admin to approve this and the follow-up PR and we're good.
Quick question, will these changes show up in the 0.9.99 test build we are promoting as well? If not, could we get a PR for that branch as well so the experience is consistent in our tester population?

Hey guys, I take it we just need the repo admin to approve this and the follow-up PR and we're good.

@CraigSellars has approved the change (see couple of comments up)

Quick question, will these changes show up in the 0.9.99 test build we are promoting as well?

These changes will show up in <IP_ADDRESS> test builds once:

This PR is merged
New <IP_ADDRESS> test builds are created (both @dexX7 & I have created new builds recently but trying to get to the bottom of a hash mismatch for the windows setup executables)
New test builds are uploaded to web URLs users are downloading from

If possible, I'd like to get the feature activation and send-all transaction type into the next preview, because both seem to be ready, or almost ready, and more exposure would be great.
This isn't required though, and I'd be fine with another preview before that, too.
Unfortunally I don't see many options to resolve the issue with the mismatching installers, so I tend to just exclude the installers, and simply publish the other files (which match perfectly).

The new hardcoded descriptions currently break the tests, which is really aweful. I'm going to push a quick fix for OmniJ, and once that was adopted, I'm fine with merging this PR.

When you're done with the tests, push just this change so I can send screen caps to the exchange partners and prompt them to push their changes. I will start promoting this test build then on those exchanges to make the jump. We should try to get the two features you mention in for next week's preview.

@zathras-crypto: quick note regarding the old previous discussion: I think we should add http://, if that's okay with you. And maybe add a . at the end of the property description.

@patrickdugan: can you please clarify what you mean with:

and prompt them to push their changes

?
I do not recommend to run the test builds in production environments, and if there is any risk an exchange does it nevertheless, we are basically forced to include the outstanding consensus affecting changes in the next preview.
Somewhat related information: we are usually able to build and compare preview builds within a day.

I will start promoting this test build then on those exchanges to make the jump.

Just for 100% clarity, this change is for 0.0.10 (currently being previewed as <IP_ADDRESS>) - we would not want any exchanges to "make the jump" to one of our 0.0.10 preview builds, they are not release-ready yet.

If possible, I'd like to get the feature activation and send-all transaction type into the next preview, because both seem to be ready, or almost ready, and more exposure would be great.

Agreed, they are 99% ready so it seems prudent to include them.

Unfortunally I don't see many options to resolve the issue with the mismatching installers, so I tend to just exclude the installers, and simply publish the other files (which match perfectly).

The installers are beneficial, and since these are preview builds perhaps it's OK that we have a non-deterministic element?  Also to remember is (for whatever reason) I can create build after build with the same setup hash on my Gitian VM.

...push just this change so I can send screen caps to the exchange partners and prompt them to push their changes....

Should we not be organizing this a little more?  Eg a PR effort; corresponding blog posts, reddit posts etc etc - given OmniChest.info is the "goto" block explorer for Omni that is where most people are going to get visibility into the MSC -> OMNI rebrand, so as mentioned yesterday at the meeting changes should be made there too.
TL:DR; rather than just "pulling the pin" as it were, perhaps better to structure the change and include as many stakeholders as we can?  IMO the 'change' should go 'live' at the same time for:

PR effort (blog, reddit, skype groups etc etc)
Block explorer (OmniChest)
Web wallet (OmniWallet)
Exchanges (as many as we can agree scheduling with)

Just my 2 cents :)

@zathras-crypto: quick note regarding the old previous discussion: I think we should add http://, if that's okay with you. And maybe add a . at the end of the property description.

Sure dude, will do :)

screen caps

As requested, but it's critical to note these are screen caps from a development version of Omni Core, and intergrators should not go and run this code :)

FYI @zathras-crypto and @patrickdugan (as I commented on Dexx's OmniJ PR https://github.com/OmniLayer/OmniJ/issues/99)
Although the Goal of OmniJ release 0.3 is to provide an RPC client and transaction builder that works with 0.0.9 and 0.0.10, the integration tests (RegTest and Consensus) are focused on 0.0.10 only (mostly)  These priorities are simply based upon what people are using OmniJ for. The only thing that I know of that is being tested is 0.0.10. Whereas the other OmniJ functions are being used by multiple projects that are currently working with 0.0.9.
I'm open for feedback on these priorities -- especially if there are use cases that I'm not aware of.

@zathras-crypto: I share your sentiment, and I think it would be helpful to lay out a plan.
I'm more than split regarding:

The installers are beneficial, and since these are preview builds perhaps it's OK that we have a non-deterministic element? Also to remember is (for whatever reason) I can create build after build with the same setup hash on my Gitian VM.

While we are not yet at a point, where it's very user friendly to verify deterministic builds, I believe breaking the approach, of having at least two matching results, undermines the whole process almost completely. So I tend to believe it would be better to leave out the non-matching elements (and of course, ideally resolving the initial issue of the mismatches).
@msgilligan: thanks for your clarification!

I believe breaking the approach, of having at least two matching results, undermines the whole process almost completely

That's a very fair point, and I don't think we'll lose many potential testers due to the lack of a full windows installer (it's not like we have to install a bunch of dependencies, just extracting the QT binary is all that's needed).  OK, so we'll do the next preview without the windows installers in the case that we're not able to resolve whatever the issue is prior to building.
@msgilligan thanks, you may be interested in looking at consensus testing via consensus hash in OmniJ when you have spare time as it should simplify the process substantially.  Support via RPC (omni_getcurrentconsensushash) is already in the current 0.0.10 codebase and the next version of the OmniChest API has support for it.  Not sure how that fits into your priorities but let me know if I can help at all :)
While on the topic @dexX7 what do you think about expanding the scope of the consensus hash to include additional state attributes?  If so how far should we go?  For example to confirm SPs do we include all properties with their metadata, or is it sufficient to just include the next_spid and next_test_spid vars?  I'd also like to include data from all other values that affect the current state such as open positions (DEx offers/accepts, open MetaDEx trades, crowdsales etc) since that is all data that can be used to actually verify consensus of the state (rather than verifying consensus of the balances which only infers consensus of the state).

@zathras-crypto: yes, we should expand the scope of the consensus hash.
A few notes here, first, let's make all consensus hashes unique:

currently we generate each consensus hash with the tally for a specific block
however, if two or more tally-collections are similar, then so are their hashes
this means there might very well be more than one similar consensus hash
what if we combine the previous consensus hash with the tally hash?
say: consensus_hash[block n] = Hash(consensus_hash[block n-1] + tally_hash[block n])
for the very first hash, there won't be a predecessor, so we'd need a filler
this is probably tricky during reorgs!

Expand the tally hash to operations, and not results:

currently we look at the results at the end of the block
but we basically don't care what happens inbetween
there might be similar balances, but with a different history
e.g. 10 MSC + 15 MSC == 7 MSC + 18 MSC
what if we expand the consensus hash on each balance changing operation?
this is much more tricky, because we batch-restore balances via persistence as sum, and not as single operations

Then you already mentioned:

we could also cover other values
such as: transactions, smart properties, open orders, ...
while transactions are somewhat easy to represent, we'd need a representation for these other items

I would like all this very much, but the more complex the consensus hashing becomes, the more pitfalls there probably are. It would also be great, if the consensus hash generation isn't intrusive, e.g. I wouldn't like to hijack to very low tally operations like CMPTally::updateMoney.

this means there might very well be more than one similar consensus hash
what if we combine the previous consensus hash with the tally hash?
for the very first hash, there won't be a predecessor, so we'd need a filler
this is probably tricky during reorgs!

I'm not sure about this one - it boils down to whether consensus hashes should be atomic or chained.  If atomic then the state can be verified without knowledge of past history, if chained then we must know consensus hash for the previous block in order to verify current, and in order to verify the previous consensus hash we need the hash for the block before that, and so on working backwards.
I can see pros and cons to both approaches - do you have a view here?
Note, if the objective is purely to make the hashes unique, adding the blockhash to the hash would serve the same purpose without requiring prior knowledge if we feel atomic consensus hashes are desired.

Expand the tally hash to operations, and not results:

Just a thought, but perhaps there is a half-way option - if per block isn't granular enough, why not expand the hash to per transaction.  If the hash was expanded to cover much more of that state then any transaction that changed any aspect of the state in a transaction would result in a different hash after that transaction.  I raise it as that would require very minimal changes I think?

I would like all this very much, but the more complex the consensus hashing becomes, the more pitfalls there probably are

Yeah, I was thinking to keep the changes within GetConsensusHash() which we can call when we decide to (on operation, on transaction, on block etc)...   But I do see your point, and we shouldn't go crazy hehe :)

@zathras-crypto I'm definitely interested in using checking consensus hashes in the tests. I should also be able to write a test that verifies the consensus hashes, right? Is the algorithm documented somewhere?

@sean FYI (& you can thank @dexX7 for the better commenting, my commenting wasn't that clear when I originally wrote the function hehe):
/**
 * Obtains a hash of all balances to use for consensus verification and checkpointing.
 *
 * For increased flexibility, so other implementations like OmniWallet and OmniChest can
 * also apply this methodology without necessarily using the same exact data types (which
 * would be needed to hash the data bytes directly), create a string in the following
 * format for each address/property identifier combo to use for hashing:
 *
 * Format specifiers:
 *   "%s|%d|%d|%d|%d|%d"
 *
 * With placeholders:
 *   "address|propertyid|balance|selloffer_reserve|accept_reserve|metadex_reserve"
 *
 * Example:
 *   SHA256 round 1: "1LCShN3ntEbeRrj8XBFWdScGqw5NgDXL5R|31|8000|0|0|0"
 *   SHA256 round 2: "1LCShN3ntEbeRrj8XBFWdScGqw5NgDXL5R|2147483657|0|0|0|234235245"
 *   SHA256 round 3: "1LP7G6uiHPaG8jm64aconFF8CLBAnRGYcb|1|0|100|0|0"
 *   SHA256 round 4: "1LP7G6uiHPaG8jm64aconFF8CLBAnRGYcb|2|0|0|0|50000"
 *   SHA256 round 5: "1LP7G6uiHPaG8jm64aconFF8CLBAnRGYcb|10|0|0|1|0"
 *   SHA256 round 6: "3CwZ7FiQ4MqBenRdCkjjc41M5bnoKQGC2b|1|12345|5|777|9000"
 *
 * Result:
 *   "3580e94167f75620dfa8c267862aa47af46164ed8edaec3a800d732050ec0607"
 *
 * The byte order is important, and in the example we assume:
 *   SHA256("abc") = "ad1500f261ff10b49c7a1796a36103b02322ae5dde404141eacf018fbf1678ba"
 *
 * Note: empty balance records and the pending tally are ignored. Addresses are sorted based
 * on lexicographical order, and balance records are sorted by the property identifiers.
 */
uint256 GetConsensusHash() {

@msgilligan: @zathras-crypto copied the relevant part from here, and the unit test may be useful in this context:

src/omnicore/rules.cpp#L435-L507
src/omnicore/test/checkpoint_tests.cpp
src/omnicore/rpc.cpp#L1561-L1593

I can see pros and cons to both approaches - do you have a view here?

You have a very good point. I've never considered atomic consensus hashes as desirable, but actually, when thinking about it, it has indeed benefits, especially because we even restore state snapshots. On the other hand, at least now, we use the consensus hashes to checkcorrect behavior, to pin down issues, and having the whole history "baked in" would cover "everything".

@msgilligan @dexX7 I made some changes to consensus hashing here https://github.com/OmniLayer/omnicore/pull/226 based on our discussions.  I still haven't put in a "predecessor" hash yet, still unsure which way to go here.


Fix copy to clipboard button style when the svg name is too long

[ ] The name should always be a single line.  It can be abbreviated with ....
[ ] The copy icon size should always remain the same.

Please, assign to me!

Please, assign to me!

Do you have any progress on this issue? @LuizHAP

I can fix that issue, if no update from LuizHAP

I can fix that issue, if no update from LuizHAP @aykutkardas

I can fix that issue, if no update from LuizHAP @aykutkardas

Please, assign to me!

Do you have any progress on this issue? @LuizHAP

I can fix that issue, if no update from LuizHAP @aykutkardas


Code Action: Adds Missing 'return' to Parser Rule

Closes one of the points of #442.
Adds return T to rule for general error

The type 'T' is already explicitly declared and cannot be inferred.

From https://github.com/langium/langium/issues/449.

There are a few cases that get mixed up in here which need a little more work to be separated cleanly. Missing returns & invalid returns are coupled together a bit, and invalid returns is in-turn coupled along with code that handles invalid infers. Plan is to come back to this at the start of Oct at the top of my list 📚 , and to clean this up a bit.

Alright, reworked this and it's now down to:

code action to add missing 'returns'
code action to correct invalid 'infers' (added due to some proximity)
same tests as before

Shaves down some of the changes for InvalidInfers by passing along the range of the appropriate CST node.

@montymxb
Looks really nice! Is the same for Actions already planned.

@dhuebner thanks! The original plan was to take care of the cases for parser rules, but it would be ideal to set that up for actions too. I believe once we have the code actions we want for parser rules, the same approach can be adapted to actions as well.


[units/urb-persister-elasticsearch] Implement elasticsearch persister

Considering the capabilities of SASI indexes in Cassandra, the need for elastic search is diminished. If necessary, this can be reopened at a later point, but since it is not worked on, I am closing the issue.


Fix ProtoBuf crash when running from_tensor_slices on large datasets

Protobufs have a hard limit of 2GB transfer. In our current dataset (with sufficient augmentation) the tensor dataset size gets to about 2.6GB and hence crashes.
Either figure out a way to load dataset via several calls or else reduce dataset size (e.g. by doing augmentation on the fly) - else use GCS (undesirable).

Fixed by reducing dataset footprint by doing data augmentation (mainly sub-trajectory sliding window augmentation) on the fly with tf.data.Dataset.window(). Can now perform max window augmentation (with a shift of 1) without any memory errors because dataset size is kept constant and window augmentation is done on the fly. Training time with these changes is no worse than before, mainly due to multi-threading of data aug steps + prefetching.