Identify sources for improving deployment time

As recent as December 7, 2016, deployment on CI could pass in ~15 minutes. Even with #994, deployment still takes ~36 minutes. We should profile and identify the most time-consuming tasks and try to refactor them where possible.

On my last deployment of 6 machines (3 masters, 3 nodes) on top of Ubuntu 16.04 cloud image it took 19 minutes. Out of these, nearly 4 minutes were on bootstrapping (installing python and pip). Maybe having an immutable image building process for the supported OSs which leaves all required packages and containers inside (based on the same ansible roles Kargo already has) could help.

Task #
Play
Task
Start time [s]
Task duration [s]
Accum duration [s]

8
k8s-cluster:etcd:calico-rr
bootstrap-os : Bootstrap - Install python 2.x and pip
4.038
234.187
234.187

723
k8s-cluster
download : Download containers if pull is required or told to always pull
592.279
70.293
304.48

390
k8s-cluster
download : Download containers if pull is required or told to always pull
445.914
27.533
332.013

156
k8s-cluster:etcd:calico-rr
docker : ensure docker packages are installed
333.295
24.95
356.963

1114
k8s-cluster
download : Download containers if pull is required or told to always pull
940.211
19.446
376.409

109
k8s-cluster:etcd:calico-rr
kubernetes/preinstall : Install packages requirements
288.212
15.954
392.363

167
k8s-cluster:etcd:calico-rr
docker : Write docker dns systemd drop-in
372.299
10.049
402.412

106
k8s-cluster:etcd:calico-rr
kubernetes/preinstall : Install latest version of python-apt for Debian distribs
278.242
9.624
412.036

1089
k8s-cluster
download : Download containers if pull is required or told to always pull
911.172
9.154
421.19

1139
k8s-cluster
download : Download containers if pull is required or told to always pull
977.277
9.046
430.236

897
k8s-cluster
download : Download containers if pull is required or told to always pull
760.823
8.754
438.99

154
k8s-cluster:etcd:calico-rr
docker : ensure docker repository is enabled
325.122
8.028
447.018

1401
kube-master
kubernetes/master : Write kube-apiserver manifest
1092.274
6.661
453.679

507
k8s-cluster
etcd : Refresh config - Create etcd config file
524.072
6.292
459.971

1453
k8s-cluster
dnsmasq : Check for dnsmasq port (pulling image and running container)
1125.897
5.743
465.714

1413
kube-master
kubernetes-apps/network_plugin/weave : Weave - wait for weave to become available
1102.563
5.627
471.341

166
k8s-cluster:etcd:calico-rr
docker : Write docker dns systemd drop-in
366.7
5.599
476.94

1022
k8s-cluster
download : Download containers if pull is required or told to always pull
848.956
5.529
482.469

997
k8s-cluster
download : Download containers if pull is required or told to always pull
826.754
5.076
487.545

922
k8s-cluster
download : Download containers if pull is required or told to always pull
787.186
4.895
492.44

1047
k8s-cluster
download : Download containers if pull is required or told to always pull
872.111
4.794
497.234

791
k8s-cluster
kubernetes/secrets : Gen_certs - update ca-certificates (Debian/Ubuntu/Container Linux by CoreOS)
713.398
4.111
501.345

748
k8s-cluster
download : Download containers if pull is required or told to always pull
681.45
4.104
505.449

428
k8s-cluster
etcd : Gen_certs - update ca-certificates (Debian/Ubuntu/Container Linux by CoreOS)
493.403
3.766
509.215

1178
k8s-cluster
network_plugin/calico : Calico - Copy cni plugins from hyperkube
1013.283
3.172
512.387

153
k8s-cluster:etcd:calico-rr
docker : ensure docker repository public key is installed
322.266
2.856
515.243

1179
k8s-cluster
network_plugin/calico : Calico - Copy cni plugins from calico/cni container
1016.455
2.673
517.916

24
k8s-cluster:etcd:calico-rr
setup
242.899
2.662
520.578

990
k8s-cluster
download : set_fact
816.24
2.426
523.004

981
k8s-cluster
download : downloading...
809.344
2.354
525.358

993
k8s-cluster
download : set_fact
821.394
2.32
527.678

68
k8s-cluster:etcd:calico-rr
kubernetes/preinstall : check resolvconf
262.271
2.273
529.951

This was the output of ansible, with some parsing. I only included tasks that took more than 2 seconds for the table, which accounts for 8.8 minutes of the ~19 total minutes, sorting task by duration. Distribution is relatively long tailed, with plenty of tasks taking less than a second. If you consider only tasks that take more than a second, you account for ~11 minutes of the 19 minutes. So there is plenty of very short lived tasks that add nearly half of the overall time.
Low hanging fruits are of course having the starting image pre-loaded with python, pip and all required containers. That would probably get rid of nearly half the time taken to deploy.

@pcm32 #1 is really just a poor choice of default image if you care about time. A minimal install of Ubuntu is going to have the slowest deploy time because installing python is a big task. In this case, your total time should be 15 minutes, not 19.
Secondly, docker images can't be pulled any faster, except by putting them in a local location in advance and modifying config. We're downloading about 800mb of data here. The ~3 mins spent downloading is about as fast as anyone could hope.
That makes 7 minutes of your 19 untouchable for all intents and purposes.
The next task in order that shows up that doesn't fall into that category is docker : Write docker dns systemd drop-in. It's not clear why it takes 10s here. It is a very small template that is easy to evaluate. The system might be having some cleanup tasks after installing docker package here, slowing down the task. There's lot a whole lot to be done here too.
Lastly, @pcm32 you hit on a big clue here. 57% of tasks take <1s. Maybe we don't need so many and maybe some can be combined. There are three areas we can approach this problem:
1 - Unused roles in Ansible play: even when you don't set vault as the script type, this role gets evaluated.. at least in my env. All tasks are skipped, but it wastes time (37s on CI). We could modify the playbook to include the role name as a var, rather than using a when conditional
2 - Clean up old pre_upgrade tasks that are older than 1 tag. I have been supporting upgrades since version tag v2.0.0. In reality, nobody has old environments of Kargo and this is extra baggage. We should provide tag-to-tag upgrades and drop any unnecessary upgrade tasks. This could save up to 36s.
3 - Move some selectors to defaults: We do some complex logic to select certain paths based on different conditions. This is because some config files live in different locations, depending on which cloud provider we have. We may save a handful of seconds here.
So, as far as low hanging fruit goes, in Ansible code changes, we can drop 1-1.5 minutes here.
The biggest difference would be if you could create an image that ran the following commands:
sudo apt-get -y dist-upgrade
sudo apt-get install -y python python-apt python-pip
curl -sSL https://get.docker.com/ | sh
docker pull quay.io/coreos/hyperkube:v1.5.3
docker pull quay.io/coreos/etcd:3.0.6
docker pull calico/ctl:v1.0.2
docker pull calico/node:v1.0.2
docker pull calico/cni:v1.5.6

I hope this doesn't come to you as any bad news, but this task gets harder each time as we add more and more features. There are many tasks not discussed here that were optimized for scale deployment, and I believe there is some tradeoff in single node deployment vs 200 node deployment, where single node takes 2-3 mins more, but 200 node deployment takes hours less than it would otherwise.

For those of you interested in deploy times for scale, here are 2 play recaps from 153 node deployment. Total time is 65 mins. The first is etcd+kube-master play. The second is kube-node play:
Note that this deploy doesn't include https://github.com/kubernetes-incubator/kargo/pull/1111
etcd+kube-master play:
11:36:27 Friday 03 March 2017  11:36:27 +0000 (0:00:00.019)       0:10:35.735 **********
11:36:27 ===============================================================================
11:36:27 etcd : Check_certs | check if all certs have already been generated on first master -- 48.54s
11:36:27 etcd : Check_certs | check if all certs have already been generated on first master -- 48.31s
11:36:27 etcd : Check_certs | check if all certs have already been generated on first master -- 47.80s
11:36:27 kubernetes/secrets : Gen_tokens | generate tokens for node components -- 46.00s
11:36:27 kubernetes/secrets : Check_certs | check if the certs have already been generated on first master -- 45.59s
11:36:27 etcd : Pre-upgrade | check if member list is non-SSL ------------------- 31.70s
11:36:27 etcd : Pre-upgrade | check if member list is non-SSL ------------------- 31.26s
11:36:27 kubernetes/secrets : Gen_certs | run cert generation script ------------ 25.31s
11:36:27 download : Download containers if pull is required or told to always pull -- 23.21s
11:36:27 etcd : wait for etcd up ------------------------------------------------ 15.49s
11:36:27 docker : ensure docker repository public key is installed -------------- 11.32s
11:36:27 etcd : reload etcd ----------------------------------------------------- 10.38s
11:36:27 docker : ensure docker packages are installed -------------------------- 10.34s
11:36:27 docker : Docker | pause while Docker restarts -------------------------- 10.03s
11:36:27 dnsmasq : Check for dnsmasq port (pulling image and running container) --- 8.40s
11:36:27 download : Download containers if pull is required or told to always pull --- 8.26s
11:36:27 kubernetes/master : Master | wait for the apiserver to be running ------- 6.37s
11:36:27 kubernetes/master : Master | wait for kube-scheduler -------------------- 5.38s
11:36:27 kubernetes/preinstall : Install packages requirements ------------------- 4.60s
11:36:27 download : Download containers if pull is required or told to always pull --- 4.43s

kube-node play:
Friday 03 March 2017  12:32:55 +0000 (0:01:17.491)       0:55:24.170 ********** 12:32:55 kubernetes/preinstall : Hosts | populate inventory into hosts file ----- 77.49s
12:32:55 download : Download containers if pull is required or told to always pull -- 72.82s
12:32:55 etcd : Check_certs | check if all certs have already been generated on first master -- 49.04s
12:32:55 kubernetes/secrets : Check_certs | check if the certs have already been generated on first master -- 46.40s
12:32:55 kubernetes/secrets : Gen_tokens | generate tokens for node components -- 43.62s
12:32:55 etcd : Pre-upgrade | check if member list is non-SSL ------------------- 31.24s
12:32:55 docker : ensure docker packages are installed -------------------------- 28.30s
12:32:55 download : Download containers if pull is required or told to always pull -- 24.62s
12:32:55 etcd : Gen_certs | run cert generation script -------------------------- 22.35s
12:32:55 download : Download containers if pull is required or told to always pull -- 20.08s
12:32:55 kubernetes/preinstall : Hosts | populate inventory into hosts file ----- 18.31s
12:32:55 download : Download containers if pull is required or told to always pull -- 14.57s
12:32:55 download : Download containers if pull is required or told to always pull -- 13.80s
12:32:55 kubernetes/preinstall : Remove search/domain/nameserver options -------- 12.04s
12:32:55 kubernetes/preinstall : Install packages requirements ------------------ 11.96s
12:32:55 download : Download containers if pull is required or told to always pull -- 11.10s
12:32:55 docker : Docker | pause while Docker restarts -------------------------- 10.03s
12:32:55 kubernetes/secrets : Gen_certs | update ca-certificates (Debian/Ubuntu/Container Linux by CoreOS) --- 9.85s
12:32:55 etcd : Gen_certs | update ca-certificates (Debian/Ubuntu/Container Linux by CoreOS) --- 9.59s
12:32:55 network_plugin/calico : Calico | Configure peering with route reflectors --- 8.93s

Thanks @mattymo!!
You could pull the containers (the ~800 MB that you mentioned) when you create the immutable image, and then leave them on the image, that would spare the download on each machine when deploying. People can then create their own immutable images with some level of frequency if we provide the template (with packer for instance, which could use our current ansible roles).
If Ubuntu cloud image is not a good choice, what is the preferred choice? A particular CentOS image? It would be good to indicate that on the documentation.

@pcm32 Ubuntu is a good choice, he was only saying that an Ubuntu-minimal (without python) isn't the best case for speed.
How packer working? I think it would be a good addition to provide image on each Kargo releases and script to build them.

My wrapper app for fuel-ccp-installer has packer templates you can have, located here:
https://github.com/openstack/fuel-ccp-installer/blob/master/doc/PACKER.md
https://github.com/openstack/fuel-ccp-installer/tree/master/utils/packer
I am on holiday part of this week, but next week I can move them to kargo contrib/ dir next week for sure.
All we need on kargo side is a hashicorp Atlas account to host some images for release. I don't think Github can be used to host ~4gb packer images.

@ant31 packer works well when tasks for node are not too complex. Also we will need additional wrappers and default variables which for example will change state for tasks like service states, mounts and so on depending on regular/packer run.
We use packer in our production environment to prepare AMIs for autoscale configuration. Kubernetes cluster-autoscaler then changes desired number of nodes depending on unscheduled pods.


Update SC dialog to save/read file structure to/from schema compare file

This PR fixes #20332.
With latest STS changes (https://github.com/microsoft/sqltoolsservice/pull/1878 and https://github.com/microsoft/sqltoolsservice/pull/1952), user can now save file structure into the schema compare file and read the stored information from the scmp file and present it on the schema compare dialog.

Found this bug while testing: https://github.com/microsoft/azuredatastudio/issues/22728

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

4 of 9 (44.44%)  changed or added relevant lines in 2 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.01%) to 41.665%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

extensions/schema-compare/src/schemaCompareMainWindow.ts
0
1
0.0%

extensions/schema-compare/src/dialogs/schemaCompareDialog.ts
4
8
50.0%

Files with Coverage Reduction
New Missed Lines
%

extensions/machine-learning/src/common/processService.ts
2
81.48%

Totals

Change from base Build<PHONE_NUMBER>:
0.01%

Covered Lines:
30359

Relevant Lines:
67464

💛  - Coveralls

Also bump the azdata dependency to 1.44 in schema compare's package.json since this depends on changes made in mssql

Done.


Ignore ServiceMonitor in bundle.yaml

Users installing the bundle.yaml most likely do not yet have the
ServiceMonitor CRD available in their cluster, therefore having this
object in the bundle.yaml would break them.
/cc @brancz @aditya-konarde

lgtm :+1:

I agree, thanks @metalmatze


Fix IPsec support for --devices

The first two commits are refactoring commits. The third fixes the support for --devices in IPsec.
Fix a bug that could cause local packet delivery to be skipped, leading to lower performance, when IPsec was enabled and `--devices` provided.

/test

/test

/test

/test


Global Search not working when folder name contains brackets: Error parsing Glob... invalid range

VSCode Version: 1.23.1
OS Version: Windows 10

Steps to Reproduce:

name a source code folder like DEV [JKR-25]. Very important, must use brackets and this format [LETTERS-numbers]
on Visual studio code open that folder
do any global search, for example try to find jquery
below the search field an error appears
Error parsing glob '!/repo/DEV [JKR-25]/**/.git': invalid range 'R' > '2'

It seems that the folder name is being use as part of a regular expression.

Does this issue occur when all extensions are disabled?: Yes

Please try Insiders, it should work there.


JSONField for BaseLogEntry data_json

This issue is very much related to the issue https://github.com/wagtail/wagtail/issues/5280 which is requesting JSONField for Blocks. I would extend that request to also cover BaseLogEntry data_json, so that it would be using JSONField as well.
My proposed solution and rationale would be pretty much the same as explained in https://github.com/wagtail/wagtail/issues/5280:
"I would like to propose a configuration option to use JSONField instead of text field which will allow the data to be stored as JSON directly.
It will also enable user to leverage query options available for the JSONField."
JSONField is now supported by Django:
https://docs.djangoproject.com/en/3.1/ref/models/fields/#django.db.models.JSONField
The current implementation is based on TextField which strangely stores empty values differently depending on the action:

Not really a huge issue but would make sense to use the same empty value, for example {}, no matter whether the implementation is based on TextField or JSONField.

Hey @m5seppal, thank you for this!
I would absolutely love to switch to JSONFIeld in both cases. We cannot yet since JSONField is a Postgres backend only feature for Django < 3.1.

Could we do it so that it's configuration based and defaults to TextField for now so people using Django < 3.1. won't have to do anything?

@m5seppal I don't think there's any good way to make that work with Django migrations - in order to change the field from one type to another, you need a migration inside the wagtail.core app. If we leave that choice up to the user, then it's not possible to ship that migration as part of a Wagtail release - and if the migrations we ship with Wagtail aren't consistent with the expected database state (as determined by the model definitions), you end up in a situation where a user running ./manage.py makemigrations ends up creating spurious 'auto' migrations inside the Wagtail package.


Removes GCP Invoker dependency on AbstractSpringFunctionAdapterInitializer

Removes dependency on AbstractSpringFunctionAdapterInitializer based on discussion in #955
This way the GCP adapter is inline with the implementation for AWS/Azure.

Hold off on reviewing this. There is an issue at deployment time.


Error in quarto blog

Hi, thanks for this extension!
I was trying to implement an image gallery using this on a page on my quarto-blog.
I repeatedly ran into the following error:
Error running filter _extensions/mcanouil/masonry/masonry.lua: _extensions/mcanouil/masonry/masonry.lua:2: attempt to call a nil value (field 'is_format')
Is there a conflict regarding the "format: html" part of the YAML? This is probably overwritten by the quarto blog framework.

As stated in the readme, this is highly experimental and might not be up to date with Quarto CLI extension API.

Thanks for the quick reply. Sorry, this issue wasn't meant as complaint, I've read the note in the readme. I still wanted to give it a try and see if it works within a blog.

No worries, it's on my to1do list soon as I get a bit of spare time😉

@nucleic-acid What version of Quarto were you using when you encountered the issue?

Hi, sorry for not getting back earlier.
Apparently I have missed a few quarto updates. I was using v1.1.251
I updated to the currently stable v1.2.269 and I can run the sample code within the blog environment.

Do you have any plans to combine masonry with infinite-scroll by the same developer? Apparently it's designed to work with masonry.

I do not know this library, I need to check it out.
Even if it is not hard to add it in theory, my final goal was initially to be able to have a more flexible grid listing layout for Quarto.

Just a thanking feedback, it works well now and the layout is really beautiful!
If I may stretch your patience a bit more: have you used it to arrange many image files, like in an image gallery?
I've run into the Problem that the layout is initialized before all images are loaded.
So after ~10 properly arranged image files, the rest overlaps, as the position is calculated before the files are loaded for (still) empty image boxes of probably 0 x 0 px.
I've tried to delay initialization until after loading of all files but the raw js from the masonry documentation does not work on combination with the quarto extension.
The layout is fixed as soon as I resize the browser window after all files are loaded. But that is not possible for e.g
mobile users.

I have no idea 😅
I started to work a bit on this with the idea to combine it with a listing grid template, in the few days before I quit my last job (thus a lot of time).
Now, I have a pretty busy schedule.
I hope to work on this to better understand the JavaScript library probably the second half of December.

No worries, thanks for replying at all! It's not an important use case. I'll see if I can do anything to contribute.


Character Limit

Is there any possibility to put a character limit for rows ?
Sometimes it is override my other conky elements as you can see in the picture.

up :) or at least to just location part


Is there a way to start chrome that preserves the chrome cache?

Thanks all for the great software
I notice from other threads that each request opens a new tab like if it were in incognito. The problem is that this wont preserve the cache and will hit my CDN for each javascript & css files (higher expense). Is there a way to start pre-render that it preserves the website cache so that each file doesn't have to be re-downloaded?

This is by design since we open separate incognito contexts for each request to prevent cookies, local storage, etc from bleeding over. It would be nice to somehow share the cache between tabs but would require some work to put together, especially sharing a cache between different servers in a cluster.
Are you just worried about the higher costs from your CDN or is your rendering too slow with those extra requests to the CDN?

@Yoshi9143 how about this: https://forums.meteor.com/t/server-side-rendering-for-seo-improvement/42327/3

I think this could be built as a plugin or something to add to Prerender/chrome but isn't currently a priority here. If anyone comes up with something in the future please let us know.


Updated Column-Colors Variant Name

Updated the name of the variant of the columns block, which allows custom coloring from 'custom-color' to 'color'
Resolves: NA
Test URLs:

Before: https://main--express--adobecom.hlx.page/drafts/casey/colors/red?lighthouse=on
After: https://update-color-variant--express--adobecom.hlx.page/drafts/casey/colors/red?lighthouse=on

Codecov Report

Merging #260 (c8b6723) into stage (6bddc2f) will increase coverage by 0.11%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##            stage     #260      +/-   ##
==========================================
+ Coverage   47.04%   47.15%   +0.11%
==========================================
  Files          29       29
  Lines        8839     8839
==========================================
+ Hits         4158     4168      +10
+ Misses       4681     4671      -10

Files Changed
Coverage Δ

express/blocks/columns/columns.js
43.06% <100.00%> (ø)

... and 1 file with indirect coverage changes
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Support for .local hostname resolution?

What is the status of support for resolving .local domain names in Alpine, such as in a Kubernetes cluster?

I believe that the default Kubernetes DNS add-on settings these days does support Alpine. Can you be more specific on what the issue was? We've been using Alpine containers inside Kubernetes with the cluster.local name.


Generate Swift typealiases for Protobuf types

Fix #189

But we currently only have that prefix for Protobuf types, not for all wallet core types. Also you can always do TrustWalletCore.Type to be sure.

@alejandro-isaza protobufs does not have prefixes anymore :(
it changed from TWStellarSigningInput => StellarSigningInput


Remove allow declaration in favour of anonymous import

A neat little trick I've been using with the atomic emulation trap handler to include a dependency without unused warnings.

Can you apply the same for the cmake folder?

Done in #7


save_outcomes mode line 116

Hello, @dnguyengithub I was trying to adapt this framework to a new dataset. While pre-processing I removed the ROT and NAV_STT and changed the length parameters wherever required accordingly to 7. I am getting an error when the reshape(-1,4) operation is applied for save_outcomes in line 116 specifically for the D["inp"] one If you have any leads can you tell me the reason?

Hi, could you be more specific?

For the mode save_outcomes: the inp and tar shape are (144, 1, 1502) then the shape of np.nonzero(tar[:,0,:])[1] is sometimes like (562,) (518,) (576,) i.e., the reshape(-1,4) operation doesn't always work. What's the need for reshape(-1,4) operation and where all it's dependency is there, let me know?

Did you modify some parts of the code?
The command np.nonzero(tar[:,0,:])[1] converts the "4-hot vector" representation back to [lat, lon, cog, sog], the output should be (562,) (518,) (576,) (xxx,).

I did modify some parts, I am running it for Gulf of Mexico part but for a different dataset and there are only 7 attributes unlike 9 attribute in original code, I just removed them [ROT, NAV_STT] in all the global variables. Those are only the changes I made (in pre-processing one) but I am not sure that affects this in the main training part?

Please make sure that the output of dataset.create_AIS_dataset is "4-hot vector" representation

Any tips on how to check that?


Header and footer not showing on first page

Steps to reproduce
Tell us about your environment:

Puppeteer version: 1.6.0
Platform / OS version: Ubuntu 16.04
Node.js version: 9.9.0

What steps will reproduce the problem?
create a multi-page pdf with displayHeaderFooter: true.
What is the expected result?
header and footer should be displayed on every page
What happens instead?
header and footer not displayed on the first page
any advise? is there some option to change this? thanks a lot!

You can fix this problem without options, you only need to use CSS rules for page:
@page {
  size: A4 portait;
  margin-left: 40px;
  margin-right: 40px;
  margin-top: 60px;
  margin-bottom: 60px;
}

The size of the margin-top and margin-bottom must be enough to fit header and footer inside. In my case I have footer and header with height 40px.


Upgrade ex_cldr to 2.33.0 to fix compilation failure on elixir 1.14

Fixes this problem https://github.com/elixir-cldr/cldr/issues/182
== Compilation error in file lib/myapp/cldr.ex ==
** (CompileError) lib/cldr/plural_rules/plural_rule.ex:1: undefined function mod/2 (expected MyApp.Cldr.Number.Ordinal to define such a function or for it to be imported, but none are available)
(ex_cldr 2.32.1) ../umbrella/apps/myapp/lib/myapp/cldr.ex:1: Cldr.Backend.Compiler.before_compile/1

Works fine with elixir 1.14 👍

Thank you. The changes were made in 4931217ca7402fc85208d781b73e3a0262871909


Minor spelling fix.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 90.485%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
466

Relevant Lines:
515

💛  - Coveralls


New rule: VarCouldBeVal

This is a first implementation of a rule, as requested in #657
Note that, for now, it only looks at locally-scoped variables inside functions. I want to make it more useful than this (think, private top-level variables, properties in classes, etc.), but given that this is my first rule, I wanted to create a quick PR to get some feedback and see if I'm going down the right path.

Assignments in if statements are not allowed:
var a = 1
if (a = 2) {
    // do something
}

Is not valid, as Assignments are not expressions, and only expressions are allowed in this context
When I added some more tests (specially around shadowed variables, thanks @schalkms !!!!), I discovered that my approach was WAY too naive. Now it's a bit more complex, taking the context into account, and trying to figure out which variable is being re-assigned by walking up the parents until a matching declaration is found

The appveyor check fail seems unrelated to my change, and having to do with a timeout :man_shrugging:

Yep I restarted the build and its green now, will review it next :)

Ok, I just ran it on detekt itself and here are some false positives:
io/gitlab/arturbosch/detekt/generator/out/Yaml.kt
	fun getIndent(): String {
		var spaces = ""
		indent times {
			spaces += "  "
		}
		return spaces
	}

infix fun Int.times(function: () -> Unit) {
	var i = this
	while (i > 0) {
		function()
		i--
	}
}

io/gitlab/arturbosch/detekt/generator/collection/RuleVisitor.kt
private fun String.trimStartingLineBreaks(): String {
	var i = 0
	while (i < this.length && (this[i] == '\n' || this[i] == '\r')) {
		i++
	}
	return this.substring(i)
}

detekt-rules/src/main/kotlin/io/gitlab/arturbosch/detekt/rules/style/ReturnCount.kt
	private fun countFunctionReturns(function: KtNamedFunction): Int {
		var returnsNumber = 0
		function.accept(object : DetektVisitor() {
			override fun visitKtElement(element: KtElement) {
				if (element is KtReturnExpression) {
					returnsNumber++
				}
				element.children.filter { it !is KtNamedFunction }.forEach { it.accept(this) }
			}
		})
		return returnsNumber
	}

Please make sure that the rule does not report any of these.
We do also have the scripts/get_analysis_projects.groovy script which downloads a bunch of test projects you could run the new rule on :).

Added infix and postfix operators ++ and -- detection. Thanks for the pointers on running my rule against both detekt and all the other projects.
I'm happy to tell you that there are no positives in detekt, and that it actually found a couple of true positives in some of the others (and no false positives!) :tada:


⚠ Bofrost.be has degraded performance

In 8d0c0fd, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 1320 ms

Resolved: Bofrost.be performance has improved in 149ac05 after 19 minutes.


BZ1241358 SingleSessionCommandService Interceptor stack is lost when …

…rollback

test case

fix applied in drools, test case from this PR merged with small enhancements to check all interceptors and their order. thanks again @tkobayas


Support Swift Package Manager

We should support SwiftPM.
But, per this thread, we can't use it because of some issue with the libxml2 module map.

I think Xcode 11.4 already fix the modulemap for libxml2. It’s time to support SPM.
See: https://developer.apple.com/documentation/xcode_release_notes/xcode_11_4_release_notes

Thank you for pointing this out! I'll investigate the possibility.


If in deleted stage, its ready to be deleted (send email at least a week before the study is permanently deleted)

Duplicate #423


Improve caching for the Catalog.getPageIndex method (PR 13319 follow-up)

This method is now being used a lot more, compared to when it's added, since it's now used together with scripting as part of the PDFDocument.fieldObjects parsing (called during viewer initialization).
For /Page Dictionaries that we've already parsed, the pageIndex corresponding to a particular Reference is already known and we're thus able to skip all parsing in the Catalog.getPageIndex method for those cases.

/botio test

Thank you for improving this!

Thank you for improving this!

As always, thanks for reviewing/merging!

Since 2021 ends in a few hours, I'd also like to wish you a happy new year!

Happy new year to you as well :-)


Filter does not work when columns hidden per Buttons API

It seems the PugIn is not capable of applying filters when some columns are hidden via Button API.
It seems to appear more often when you hide columns and then reorder some other columns.
jquery.dataTables.yadcf.js:654 Uncaught TypeError: Cannot read property 'length' of undefined
at escapeRegExpInArray (jquery.dataTables.yadcf.js:654)
at yadcfMatchFilterString (jquery.dataTables.yadcf.js:775)
at exFilterColumn (jquery.dataTables.yadcf.js:3998)
at Object.exFilterExternallyTriggered (jquery.dataTables.yadcf.js:4367)
at executeFilter (dataTables.js:612)

Please provide a link to a jsbin/jsfiddle with a test case

Would a link to a demo-system also be okay? Would be much easier for me right.

yes, but I might ask to link to a yadcf in my dropbox so I will be able to debug it

that souldn't be a problem. i will send you a link and login via email soon.

fixed in 0.9.2.beta.12


New flag not created, old flag unresolved if flag reason matches user's previous flag for taxon

From: https://forum.inaturalist.org/t/flags-mysteriously-unresolving/52650/

Find a taxon which you have already flagged, and make sure that flag has been resolved.
Flag that taxon again, and make sure the "Reason" field matches the "Reason" field from your resolved flag.
New flag is not saved, old flag gets unresolved.

resolved by #4157


Use Favicon Color on Sandbox Banner - Option Not Working

Describe the bug
In extension version 1.26, the option in the settings for 'Use favicon color on sandbox banner' is off, however the custom favicon color is still being applied to my sandbox banner.
To Reproduce
Steps to reproduce the behavior:

Go to a sandbox org
In the extension options, set a favicon color, i.e. LimeGreen.
Turn off "Use favicon color on sandbox banner"
Reload sandbox org page
See the banner is green.

Expected behavior
The color of the banner is not changed unless the "Use favicon color on sandbox banner" is enabled.
Screenshots

Desktop:

OS: Win11
Browser: Chrome
Version: 130.0.6723.70

Hi @Garywoo,
Thanks for raising this issue, I'll take a look !


404 returned insted of 403 when uisng the dotfiles with "deny option"

hi
when i used the below code with express i resived 404 insted of 403 but the documnetation say it should return 403
js app.use(express.static('public',{     'dotfiles' : 'deny' }));

Hello, and very sorry for that! I used an example from the readme to add that option to and request a dotfile and got a 403. Also our test suite (https://github.com/expressjs/serve-static/blob/master/test/test.js) also shows a 403. So there must be some weried edge case you are encountering I cannot figure out.
Can you provide all of the following so I can reproduce and determine the issue?

Version of Node.js you are using
Version of this module you are using
A complete app that I can run without modification that will allow me to reproduce. Any modules besides this one used be sure to include their exact versions
The exact path to the dotfile so I can recreate it for the app
The http request to make against the app that shows a 404 instead of expected 403

Thank you!

Hi
thank you very mush for answering me i really appriciate that :+1:
1- node version is v18.12.1
2-express version  is 4.16.1
3-server-static version is 1.15.0
4-https://gist.github.com/mina20088/621d2cb80c0a0b8f44e7686ef1237a86 this is a link to the app but there are some files that needs to be createed as public folder
5- and that is part of the code which contain the request to  access a startic file
if there are any other information please send me

Thank you. The code there is a lot different from what you first posted. Specifically, fallthrough: true means that when a file is not served (through any error, even the 403s) the error will not be sent and instead the request moves to the next spot. If you want the 403s you need to turn fallthrough off to allow serve-static to send back errors regarding unable to load files.

Sorry, didn't mean to close this. I still need to test your example.

no problem thank you that you answered in all cases

by the way i tried it know and it works as you said the fallthrought needs to be false in order to get the error message on the screen think it is workin know

I think this can be closed now right?

solved


[BUG]: Missing data in sharepoint lists after running flow

Describe the bug
The latest published flow for publishing sharepoint lists doesn't include the admin config data required in CI_configAdminSetup to populate the drop down for Notification Method.
To Reproduce
Steps to reproduce the behavior:

Run SP List flow
Go to admin app
Click Admin Settings
Click Notification Method drop down
You'll see it just says "Find Items" as the required data in SP isn't there.

Expected behavior
SP list flow should load all the data like it used ot
Screenshots
If applicable, add screenshots to help explain your problem.
Interface(please complete the following information):

Browser [e.g. chrome, safari]
Version date[e.g. 2020.04.03]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional context
Add any other context about the problem here.

Hi @bhaub78,
The flow in question has not been changed since the first version. If you have deployed the lists already, you would not repeat this step again if updating. We have not changed the column structure of any lists.
Another user has experienced this. If you are re-importing the Admin app or importing it for the first time, it will have existing SharePoint connections in it. You will need to remove these again and add your own.
Once that is done, you will need to save and republish the app in order for it to run with the correct connection and show options.
For reference: https://github.com/MicrosoftDocs/powerapps-docs/issues/1139#issuecomment-599032508


travis: add 1.x to go version

Would you mind explaining what this is for?

@dlorenc Sure, 1.x means latest stable go version.
https://docs.travis-ci.com/user/languages/go/#Specifying-a-Go-version-to-use

@nkubala ping :)

@nkubala can you rebase ? i delete my branch already.

Can't rebase since the branch has already been deleted. Closing in favor of https://github.com/GoogleCloudPlatform/container-diff/pull/158


SCP: Remove ByteSliceHasher, use ShortHash instead

ByteSliceHasher was renamed in upstream commit
9c1c9f01446fdac9f8f17789a51bb925479746f0 (2019-09-22, released as part of v12.1.0),
to ShortHash, as it allowed to hash without serialization,
but wasn't removed as part of the recent SCP upgrade.
This also uncovered that SipHash was not linked in,
as ShortHash is the only file using it, and that the COPYING mention was missing.

This is pretty cut-and-dry so I'm going to go ahead with this (it's also blocking the real enhancement I was working on).


⚠ Spryr has degraded performance

In 1b6561b, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 7285 ms

Resolved: Spryr performance has improved in 2b9216b after 28 minutes.


typo on sample landing page

Under the "batteries included"...
"repetetive" should be "repetitive"...sorry could not figure out a better place to report this

Ah, thanks for reporting! Fixing :)

Fixed!


Fix double release when closing side window

isReleasedWhenClosed is YES by default for historical reason, but it needs to be set to NO with ARC, otherwise there is an extra autorelease call when closing window causing crash.

cc @dkwingsmt


Admin UI systems filters should be accessible by the corresponding roles

A non privileged admin UI user should may access jobs, services and servers filters by having the corresponding view roles.
Your pull request should…

[ ] have a concise title
[ ] close an accompanying issue if one exists
[ ] be against the correct branch (features can only go into develop)
[ ] include migration scripts and documentation, if appropriate
[ ] pass automated testing
[ ] have a clean commit history
[ ] have proper commit messages (title and body) for all commits
[ ] have appropriate tags applied

This is kinda gimmicky, but what would be the expected set of minimum roles to view just these options?  ROLE_ADMIN_UI plus the changes here?  Because that leads to a broken UI.  The test user is trapped on the events tab with the loading spinner, and no hamburger on the left to get to the other tabs...

You need the ROLE_UI_JOBS_VIEW to see especially the jobs tab. But if the user is not an global admin, the filters endpoint will be empty or hidden. Same for servers and services.
The minimum role set is:

ROLE_ADMIN_UI to generally see the admin UI parts
ROLE_UI_NAV to generally see the navigation menu
ROLE_UI_NAV_SYSTEMS_VIEW to see the systems icon/option in the navigation menu
ROLE_UI_JOBS_VIEW to see the jobs tab
ROLE_UI_SERVERS_VIEW to see the servers tab
ROLE_UI_SERVICES_VIEW to see the services tab

I've created an sample user on develop.opencast.org (username and password: systems-viewer) to test this scenario. There you can verify, that the filter pane isn't shown.


Missing figure when running Hanon

This command
stack exec -- hs-pattrans -h
gives the following output in the terminal:
Starting analysis
Parsing docs/out/hanon/ref...
Parsing hanon in Parser.hs
Parsed.
hanon:exp:pattern1
hanon:exp:pattern2
hanon:exp:pattern3
hanon:exp:pattern4
hanon:exp:pattern5
hanon:exp:pattern6
ALL {
total: 8
exact: 0.00% (0)
transposed: 0.00% (0)
tonT: 37.50% (3)
inverted: 12.50% (1)
augmented: 0.00% (0)
retrograded: 0.00% (0)
rotated: 12.50% (1)
trI: 0.00% (0)
trA: 0.00% (0)
trR: 0.00% (0)
tonrotated: 0.00% (0)
trtonA: 0.00% (0)
other: 37.50% (3)
}
There are six patterns in total in the dataset, which is correct.
But when we go into ./docs/out/hanon/ref/hanon/exp/
We only see five figures generated.

Turns out this is not a real issue.
If you check out the flexiChecker branch, you would see that the critical line of code that renders charts is commented out. Hence, you were probably viewing obsolete charts from a previous run.
So, uncommenting that and importing Analysis instead of CompoAnalysis does the job:
-- import CompoAnalysis
-- import ProtoAnalysis
-- import NewAnalysis
import Analysis
...
runAnalysis :: (Bool, Bool) -> FilePath -> IO [PatternGroup] -> IO ()
...
          renderOne pg an -- produce pie chart
...


Update website url in the README.md

changed Check the Website here.  the proper url Check the Website here.
Now the url link seems to be working.
Can I create a PR for this issue.

hii
please assign me this issue

Hii @Manice18, the link in readme is updated,,can you work more on the content and alignment in readme?

Hii @Manice18, the link in readme is updated,,can you work more on the content and alignment in readme?

Should I update the readme as a whole and make it look more clean?

Should I update the readme more as a whole and make it look more clean? Is that what you are asking me to do?

Yess @Manice18..you can start working on this..

Heyy @Manice18 are you working on this?

Yes, I am , i'll be done within tomorrow, actually I have my semester practical exams today, so I was a bit busy. But I'll complete it with today.  Just give me 1day @MonalikaPatnaik

@MonalikaPatnaik  I updated the screenshots with the current version of the website
corrected urls of the project maintainers and made them clicklabe so that on click user is taken to the github profile of the clicked maintainer's profile
fixed typos
added few points in key features
adjusted height and width of project maintainer's profile images and also tech stack icons

@MonalikaPatnaik @Mansi168 I added the screenshots, can you please verify and tell me if i am allowed to push the changes?
Yeah, you are good to go!
Thanks!

Made the pr @MonalikaPatnaik !

I am closing this issue now as the merge is complete @Mansi168 @MonalikaPatnaik


Direct port of lm-buddy to Lumigator

What's changing
lm-buddy is a Python library that we developed to be a collection of jobs for fine-tuning and evaluating open-source (large) language models. The library makes use of YAML-based configuration files as inputs to CLI commands for each job and provides a uniform interface to access different sources of data and models.
lm-buddy was initially developed to fill a need for finetuning and evaluation, and we are continuing to use the evaluation component as a core for Lumigator. However, as a result of starting on Lumigator work in parallel, we now have a fair amount code duplication. There are a number of issues we could resolve by merging lm-buddy code into Lumigator.
1. Simplifying our dependency chain: Right now the lm-buddy code is shipped as a pip dependency (i.e. for every new lm-buddy release, we build/publish a package on pypi and this is downloaded at setup time within Lumigator. ) We wouldn't need to do this anymore with lm-buddy contained within lumigator and shipped with the repo as-is.
2. Remove redundant components: we have a chance to make lm-buddy simpler and smaller as a part of this port, and as a result, take up less of a footprint within lumgiator as we ship it. We no longer use finetuning, so can remove all these components. We additionally can probably remove ragas evaluation + prometheus, but will leave this for discussion.
3. Remove redundant routes: Duplication of inference against OpenAI/Mistral for evaluation: we do this within Lumigator via ground truth and in lm-buddy via offline evaluation. We have a chance to fold these two code pieces together.
There are several pieces to this change:

A straight port of lm-buddy as part of the lumigator backend to make sure all functionality still works.  <<- this PR
1a. Investigate removing ability to use CLI with lm-buddy and call modules directly as part of simplification
Removing redundant components <-- will be addressed once this PR is approved
Removing redundant routes.  <-- will be addressed once this PR is approved
Unit tests and documentation  <-- will be addressed once this PR is approved (will need to modify tests to run lm-buddy in addition to pants test lumigator/python/mzai/backend/tests:backend_tests@parametrize=linux_cpu)

How to test it
To test this branch, pull it and run:

make local-up and rebuild Lumigator

Add a new sample dataset using the OpenAPI docs page endpoint or via cURL or via notebook helper commands.
That dataset will have a UUID that you can copy.  .

Run the following within a new Jupyter notebook connected to the local environment:

import io
import os
from pathlib import Path
from typing import Any, Dict  # noqa: UP035
from uuid import UUID
import json
import numpy as np

import pandas as pd
import requests

# APP URL
API_HOST = os.environ["LUMIGATOR_SERVICE_HOST"]
API_URL = f"http://{API_HOST}/api/v1"

# Ray URL
RAY_HEAD_HOST = os.environ["RAYCLUSTER_KUBERAY_HEAD_SVC_PORT_8265_TCP_ADDR"]
RAY_SERVER_URL = f"http://{RAY_HEAD_HOST}:8265"

# base S3 path
S3_BASE_PATH = "lumigator-storage/experiments/results/"

# EVAL_METRICS is a dict we use to show / refer to eval metrics
# in pandas dataframes and when filtering by metric name.
# The dict format is
# "column name": [list of keys to get val in nested results dict]
EVAL_METRICS = {
    "Meteor": ["meteor", "meteor_mean"],
    "BERT Precision": ["bertscore", "precision_mean"],
    "BERT Recall": ["bertscore", "recall_mean"],
    "BERT F1": ["bertscore", "f1_mean"],
    "ROUGE-1": ["rouge", "rouge1_mean"],
    "ROUGE-2": ["rouge", "rouge2_mean"],
    "ROUGE-L": ["rouge", "rougeL_mean"],
}

# the following file contains information about different models
# (currently just max RAM consumption on the TB inference task)
MODEL_INFO_FILE = "assets/model_info.csv"

def make_request(
    url: str,
    method: str = "GET",
    params: Dict[str, Any] = None,  # noqa: UP006
    data: Dict[str, Any] = None,  # noqa: UP006
    files: Dict[str, Any] = None,  # noqa: UP006
    headers: Dict[str, str] = None,  # noqa: UP006
    json_: Dict[str, str] = None,  # noqa: UP006
    timeout: int = 10,
    verbose: bool = True,
    *args,
    **kwargs,
) -> requests.Response:
    """Make an HTTP request using the requests library.

    Args:
        url (str)
        method (str, optional): The HTTP method to use. Defaults to "GET".
        params (Dict[str, Any], optional): URL parameters to include in the request.
        data (Dict[str, Any], optional): Data to send in the request body.
        files (Dict[str, Any], optional): Files to send in the request body.
        headers (Dict[str, str], optional): Headers to include in the request.
        timeout (int, optional): Timeout for the request in seconds. Defaults to 10.

    Returns:
        requests.Response: The response object from the request.

    Raises:
        requests.RequestException
    """
    try:
        response = requests.request(
            method=method.upper(),
            url=url,
            params=params,
            data=data,
            files=files,
            headers=headers,
            timeout=timeout,
            json=json_,
            *args,
            **kwargs,  # noqa: B026
        )
        response.raise_for_status()
        if verbose:
            print(f"{json.dumps(response.json(), indent = 2)}")
    except requests.RequestException as e:
        print(f"Request failed: {e}")
        raise
    return response

def experiments_submit(
    model_name: str,
    name: str,
    description: str | None,
    dataset_id: UUID,
    max_samples: int | None = None,
    system_prompt: str | None = None,
) -> requests.Response:
    if system_prompt is None and (
        model_name.startswith("oai://") or model_name.startswith("http://")
    ):
        system_prompt = "You are a helpful assistant, expert in text summarization. For every prompt you receive, provide a summary of its contents in at most two sentences."  # noqa: E501

    payload = {
        "name": name,
        "description": description,
        "model": model_name,
        "dataset": dataset_id,
        "max_samples": max_samples,
        "system_prompt": system_prompt,
    }

    response = make_request(f"{API_URL}/experiments", method="POST", data=json.dumps(payload))
    return response

 # set this value to limit the evaluation to the first max_samples items (0=all)
max_samples = 0
# or your model of choice from the choices
# https://github.com/mozilla-ai/lumigator/blob/273f3ba0ec042fbed4f532ce535c7e855d3cc862/lumigator/python/mzai/backend/config_templates.py#L64
model = 'oai://gpt-4o-mini'
dataset_id = "dbe05bc5-4956-43fb-9afc-f2df46e35cc3" # the ID of the dataset you uploaded

# now look for the dataset on lumigator
descr = f"Testing {model} summarization model on your csv"
experiments_submit(model, "lumi", descr, dataset_id, max_samples)

Related Jira Ticket
https://mzai.atlassian.net/browse/LUMI-45
Additional notes for reviewers
This will be part of a series of PRs that we approve sequentially as part of the larger change.
I already...

[ ] added some tests for any new functionality;
[ ] updated the documentation.

cc @ividal @macaab26 @javiermtorres @agpituk @davidmanzanoai

Note that tests are currently failing - will start fixing but still happy to take first-pass comments at this point.

I'd like to try running this on coreweave (ideally before the merge if you don't mind) I think the work_dir approach might need some tuning (I was getting 'request too large' error when trying it before)

Yup, works for me, let's pair if you have some time over the next few days to debug


Mic/audio playback does not work in FF 59.0.1

Mic/audio playback does not work on FF 59.0.1 or nightly build 60.0b5.
Works fine in up-to-date IE Edge and Chrome.
We tested on three separate computers. All had the same results with FF not working and Chrome/IE working correctly.
FF Console is reporting.
Media resource blob:https://lex-web-ui-codebuilddeploy-1pxeax9qz-webappbucket-.....s3.amazonaws.com/9e9845e4-bb96-404e-b4f4-6692de6a2e70 could not be decoded.
index.html
All candidate resources failed to load. Media load paused.
index.html
Media resource blob:https://lex-web-ui-codebuilddeploy-1pxeax9qz-webappbucket-....s3.amazonaws.com/9e9845e4-bb96-404e-b4f4-6692de6a2e70 could not be decoded, error: Error Code: NS_ERROR_DOM_MEDIA_METADATA_ERR (0x806e0006)
Edited URL to remove the actual bucket

Thanks for reporting the issue! This is a feedback to the Lex product team. I have reached out to the Lex team and will report back when i hear back

I have shared the feedback to the Lex team and they informed me it's on their backlog to fix. the current workaround is to use Chrome.


Navigating from the updated documentation page to working-with-elements returns a 404

Navigating from http://watir.com/updated-documentation/ to http://watir.com/updated-documentation/working-with-elements is returning a 404

Closing this and adding to the accurate repo.


Feature Request : GLM-4V-9B Model

Hi there, is it possible to add the new GLM-4V-9B model ? Thanks you
https://github.com/THUDM/GLM-4
huggingface : https://huggingface.co/THUDM/glm-4v-9b
modelscope:https://modelscope.cn/models/ZhipuAI/glm-4v-9b/summary

@elismasilva project abandoned ? No news from you for a while

@elismasilva project abandoned ? No news from you for a while
It is not abandoned, I intend to resume it, however, as I am working on another project and in addition to that my work, I am not finding time to work on more than one project simultaneously, I have many ideas for improvements beyond what you have already asked for, too. I am very anxious to be able to resume as soon as possible and continue.

@elismasilva project abandoned ? No news from you for a while
It is not abandoned, I intend to resume it, however, as I am working on another project and in addition to that my work, I am not finding time to work on more than one project simultaneously, I have many ideas for improvements beyond what you have already asked for, too. I am very anxious to be able to resume as soon as possible and continue.

ok nice! thanks a lot


Avoid base::RefCounted to be refcounted

Fixed https://github.com/ltilve/chromium/commit/1b4bfef3f715b132fada27a125e32f9d46b01608


conda-tree 1.1.0 package on conda-forge is flagged by conda-forga as possibly malicious

https://conda-forge.org/blog/posts/2023-03-12-circle-ci-security-breach/#circleci-security-incident
contains a json listing packages which potentially are malicious since they were released during the time when conda-forge's keys were compromised. I'm puzzled that this release is still available on conda-forge but this is their issue. Could you please re-release the package to ensure its security? Even if this was a false-positive and you are certain that it was not tampered with: I could not currently install conda-tree from conda-forge according to my company policies since it is blacklisted via the json from conda-forge.
Thanks!

thanks for the warning, I was not aware of this, I triggered a rebuild of the package on the conda-forge repo.

new rebuild is up.


Unclear documentation

It's been a week, I'm still trying to solve this documentation.
The resource group names are very confusing. I have three resource groups:
1- For the cluster I created
2- For the key vault
3- Auto created RG by Azure for ScaleSets.
which one is which one on this document.
SecretProviderClass object.
which value does this area expect from me?
userAssignedIdentityID: "servicePrincipalClientID"
What is a service principal client id?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: eee999c1-32dd-d014-40df-8027adee1258
Version Independent ID: 19a29c09-79f8-580a-841f-873e2801da56
Content: Integrate Azure Key Vault with Kubernetes
Content Source: articles/key-vault/general/key-vault-integrate-kubernetes.md
Service: key-vault
Sub-service: general
GitHub Login: @msmbaldwin
Microsoft Alias: mbaldwin

@fkucuk
Thanks for your feedback! We will investigate and update as appropriate.

@fkucuk
Thank you for your post! After walking through the documentation myself, I'll share my findings below.
Resource groups:

Cluster Resource Group: This resource group holds the AKS cluster that you created.
Azure Key Vault Resource Group: This resource groups holds the Azure Key Vault that you created.
Node Resource Group: This resource group was created automatically and can be seen after running "az aks show"

SecretProviderClass object section

If you want to create your own custom SecretProviderClass object with provider-specific parameters for the Secrets Store CSI driver, the YAML can be found here. - v1alpha1_secretproviderclass_service_principal.yaml

userAssignedIdentitiyID/servicePrincipalClientID:

If you created a user-assigned identity, you should be able to get the client ID through the portal.

However, if you didn't create a user-assigned identity and if the value is empty (I left it empty in my lab), it defaults to use the system-assigned identity on the VM.

Once you get passed the SecretProviderClass object section, you should be able to continue with the documentation.
I hope this helps! If you're still having issues with this, please let me know.
Thank you for your time and patience throughout this issue.

@fkucuk
I just wanted to check in and see if you had any other questions or if you were able to resolve this issue?

@James-Hamil-MSFT
Thanks for the interest. I have resolved the situation with our local MS engineers.
There are lots of concepts in this document. It is hard to digest at first. Let's go over the documentation together. I want you to know I am not an experienced AKS user.
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#use-managed-identities
Well, we are going to use "Managed Identities".
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#deploy-an-azure-kubernetes-service-aks-cluster-by-using-the-azure-cli => CHECK
principalId, clientId, subscriptionId, and nodeResourceGroup copied for later use.
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#install-helm-and-the-secrets-store-csi-driver => Nothing fancy here, CHECK
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#create-an-azure-key-vault-and-set-your-secrets => I already have one in a different Resource Group. I am going to use it.
So far so good.
But wait..
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#create-your-own-secretproviderclass-object => Well this section is a bit confusing.
I started with the oath to use Managed Identity. There are three new concepts: podIdentity, VMManagedIdentiy and userAssignedIdentityID.
There is a comment also:
[OPTIONAL] If you're using managed identities, use the client id to specify which user-assigned managed identity to use. If the value is empty, it defaults to use the system-assigned identity on the VM
Well, I have not created an identity, So I assume this is the clientId I have noted at the beginning.
Next step:
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#install-azure-active-directory-azure-ad-pod-identity
I guess I am assigning some roles to my cluster's identity. But why? And what must be the scope? Which resource group should I use? Cluster's, Azure Key Vault's or Nodes'?
RESOURCE_GROUP=contosoResourceGroup

az role assignment create --role "Managed Identity Operator" --assignee $clientId --scope /subscriptions/<SUBID>/resourcegroups/$RESOURCE_GROUP

az role assignment create --role "Virtual Machine Contributor" --assignee $clientId --scope /subscriptions/<SUBID>/resourcegroups/$RESOURCE_GROUP

Never mind, let's continue. We can always pick one of them and try another if it doesn't work, right?
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts

helm install pod-identity aad-pod-identity/aad-pod-identity

The second step of this section tells us to install the Azure AD identity. Which turns out to be the "pod identity". Wait a minute, we were supposed to use "Managed Identity". I am a newbie, so I think this must be something required.
3. Create an User-Assigned managed identity. In the output, copy the clientId for later use.
az identity create -g $resourceGroupName -n $identityName
Well here's the famous trilemma again. Which resource group should I use? Let's pick one and continue (well with the previous one, I have three more correlations to try. which makes nine attempts to figure out the right ones in the worst case.)
https://docs.microsoft.com/en-us/azure/key-vault/general/key-vault-integrate-kubernetes#use-managed-identities-1
"If you are using pod identities.." Am I, should I?
Well I know it is hard to write a walkthrough or documentation, especially if you are an expert on that subject.

I see that there are three different scenarios told here at the same time. I suggest you to separate them.
Some users may prefer to place their Azure Key Vault in different Resource Group. Using the same resource group name may be confusing.

How did I solved?
I used the default clientId of the cluster's identity. And I gave permission to this identity for reading the KeyVault.

@fkcuk I reviewed your concerns and it seems like tutorial showed several different ways to set it up with different identities. I will update document to provide steps for most common scenario. Please use product documentation which should clarify issues:
https://azure.github.io/secrets-store-csi-driver-provider-azure/getting-started/usage/#provide-identity-to-access-key-vault

#please-close


Implement application menu (main menu)

In Unix systems apps provide menu options for the OS that are shown in the main menu outside of the window render area. Need to implement a view API for this.

This might be out of scope, after Window Extensions (#77) users can get the window handle and create the menu.

This is out of scope. Developers can use system specific APIs to create the menu, if a window handle is needed use a WindowExtension to get it


Merge the Beginners and Guidelines pages

Transfer this content to the 'Guidelines' page at the bottom of page.

Should the Beginner's page link from the Navbar be removed then? If yes, assign this issue to me, I'd like to work on it.

Should the Beginner's page link from the Navbar be removed then? If yes, assign this issue to me, I'd like to work on it.

Yes, and its contents to be transferred to Guidelines.
I'm assigning you

I've made a pull request regarding the same. Please, have a look at it.

Can this issue be closed now?


Create 739-Daily-Temperatures

File(s) Modified: 0001-two-sum.py, 0002-add-two-numbers.py, etc...
Language(s) Used: python, javascript, etc...
Submission URL: _https://leetcode.com/problems/[problem-name]/submissions/xxxxxxxxx/_

Important
Please make sure the file name is lowercase and a duplicate file does not already exist before merging.

Hi, can you please provide your Leetcode submission link and rename your file using the filename convention? ( please see https://github.com/neetcode-gh/leetcode/blob/main/CONTRIBUTING.md )

File(s) Modified: 739-daily-temperatures.js
Language(s) Used: javascript
Submission URL: https://leetcode.com/problems/daily-temperatures/submissions/911085363/

Same solution as existing solution


Documentation has syntax error

Page http://mithril.js.org/mithril.html has syntax error:
Actual:
In Shorthand syntax:
var MyComponent = {
...
}

m.render(document.body, [
    //the two lines below are equivalent
    m(component, {data: "world"}),
    m.component(component, {data: "world"})
])

Expected:
var MyComponent = {
...
}

m.render(document.body, [
    //the two lines below are equivalent
    m(MyComponent, {data: "world"}),
    m.component(MyComponent, {data: "world"})
])

Seems this have been fixed by the below commit
https://github.com/lhorie/mithril.js/commit/6ffeeaa19d5590d34082c24706a42e4167cb0a15
However, the site is still not updated. Perhaps, a rebuild required!?

@sanspace There's other similar issues that have also been caught, but not updated on the main site. IIRC @lhorie rebuilds the live site on each release (major/minor/patch), so it might be waiting for the next train (which is admittedly overdue IMHO).

So, we are waiting for a train and we do know where it would take us!! :laughing:

Fixed w/ 1.0 and new site


dev: automatically-submit-and-ask-for-next-issue

dev: automatically-submit-and-ask-for-next-issue
dev: automatically-submit-and-ask-for-next-issue

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Current dependencies on/for this PR:

main

PR #100

PR #101

PR #102

PR #106

PR #107

PR #108

PR #109   👈

This stack of pull requests is managed by Graphite.

Merge activity

Jan 13, 8:02 PM: @MartinBernstorff merged this pull request with Graphite.


[Event Request] Table 7334 OnOpenItemTrackingLinesOnBeforeSetSource

Please add the same trigger OnOpenItemTrackingLinesOnBeforeSetSource from table 7332 also to table 7334:
[External] OpenItemTrackingLines()
TESTFIELD("Item No.");
TESTFIELD("Qty. (Base)");
WhseWorksheetLine.INIT;
WhseWorksheetLine."Whse. Document Type" :=
WhseWorksheetLine."Whse. Document Type"::"Internal Pick";
WhseWorksheetLine."Whse. Document No." := "No.";
WhseWorksheetLine."Whse. Document Line No." := "Line No.";
WhseWorksheetLine."Location Code" := "Location Code";
WhseWorksheetLine."Item No." := "Item No.";
WhseWorksheetLine."Qty. (Base)" := "Qty. (Base)";
WhseWorksheetLine."Qty. to Handle (Base)" :=
"Qty. (Base)" - "Qty. Picked (Base)" - "Pick Qty. (Base)";
//++++++++++++++++++++++++++++++++++++++++
OnOpenItemTrackingLinesOnBeforeSetSource(VAR Rec,VAR WhseWorksheetLine);
//----------------------------------------
WhseItemTrackingLines.SetSource(WhseWorksheetLine,DATABASE::"Whse. Internal Pick Line");
WhseItemTrackingLines.RUNMODAL;
CLEAR(WhseItemTrackingLines);

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


feat(CategoryTheory): the regular topology is subcanonical

We prove that the regular topology on any Preregular category is subcanonical.

bors r+


Verbose

Adds a verbose start option. By default it will not print the logs to the terminal.

Coverage increased (+0.001%) to 27.266% when pulling f1ca18db9de7089e42a1406f7121d1a10e83d6cf on verbose into 5ed7e41b140c0f7543ce3aa5d1bbc2b61edc5c95 on master.


feat: Updating mustache files to use bearer token/no auth rest client and request classes

Fixes
A short description of what this PR does.
Updating mustache files to use bearer token/ no auth request and rest clients in appropriate cases
Checklist

[x] I acknowledge that all my contributions will be made under the project's license
[ ] Run make test-docker
[ ] Verify affected language:

[ ] Generate twilio-go from our OpenAPI specification using the build_twilio_go.py using python examples/build_twilio_go.py path/to/twilio-oai/spec/yaml path/to/twilio-go and inspect the diff
[ ] Run make test in twilio-go
[ ] Create a pull request in twilio-go
[ ] Provide a link below to the pull request

[ ] I have made a material change to the repo (functionality, testing, spelling, grammar)
[ ] I have read the Contribution Guidelines and my PR follows them
[ ] I have titled the PR appropriately
[ ] I have updated my branch with the main branch
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I have added the necessary documentation about the functionality in the appropriate .md file
[ ] I have added inline documentation to the code I modified

If you have questions, please create a GitHub Issue in this repository.

--- Suggestion ---
If I have to think about this task.
We need to modify at apiResource level. (example, modifying creator, restclient etc)
For an apiResource, Creator, restclient etc remains the same.
We can create a hashMap in apiResource class which will store all keys that you want to use.
Populate keys for each apiResource and then directly use it in creator, reader, updater etc ....
Let me know if this works or has some drawbacks.

--- Suggestion --- If I have to think about this task.
We need to modify at apiResource level. (example, modifying creator, restclient etc) For an apiResource, Creator, restclient etc remains the same.
We can create a hashMap in apiResource class which will store all keys that you want to use. Populate keys for each apiResource and then directly use it in creator, reader, updater etc ....
Let me know if this works or has some drawbacks.

--- Suggestion --- If I have to think about this task.
We need to modify at apiResource level. (example, modifying creator, restclient etc) For an apiResource, Creator, restclient etc remains the same.
We can create a hashMap in apiResource class which will store all keys that you want to use. Populate keys for each apiResource and then directly use it in creator, reader, updater etc ....
Let me know if this works or has some drawbacks.

--- Suggestion --- If I have to think about this task.
We need to modify at apiResource level. (example, modifying creator, restclient etc) For an apiResource, Creator, restclient etc remains the same.
We can create a hashMap in apiResource class which will store all keys that you want to use. Populate keys for each apiResource and then directly use it in creator, reader, updater etc ....
Let me know if this works or has some drawbacks.

The variables/extensions we set at api resource level will not available inside sections like this
{{#headerParams.0}}
private void addHeaderParams(final {{vendorExtensions.x-http-class-prefix}}Request request) {
{{>headerParams}}

Please add an example spec in https://github.com/twilio/twilio-oai-generator/blob/main/examples/spec/twilio_api_v2010.yaml  for auth token based authentication api so that we can see how the new files are generated

Please add an example spec in https://github.com/twilio/twilio-oai-generator/blob/main/examples/spec/twilio_api_v2010.yaml for auth token based authentication api so that we can see how the new files are generated

added


PubSub: cannot cast AckReplyConsumer with 'ORIGINAL_MESSAGE' header

Since GcpPubSubHeaders.ACKNOWLEDGEMENT is deprecated, messages should not be acknowledged anymore using this peace of code, which is used also in the documentation:
AckReplyConsumer consumer = message.getHeaders().get(GcpPubSubHeaders.ACKNOWLEDGEMENT, AckReplyConsumer.class);
consumer.ack();

Now when casting the ORIGINAL_MESSAGE header there is an exception thrown, so none of these two approaches work:
AckReplyConsumer consumer = (AckReplyConsumer) message.getHeaders().get(GcpPubSubHeaders.ORIGINAL_MESSAGE);

AckReplyConsumer consumer = message.getHeaders().get(GcpPubSubHeaders.ORIGINAL_MESSAGE, AckReplyConsumer.class);

Please update the documentation and provide a solution not using the deprecated header

Our current documentation doesn't mention GcpPubSubHeaders.ACKNOWLEDGEMENT header at all. Which version of the docs are you looking at and which version of the project are you using?
Here's the documentation on how to use GcpPubSubHeaders.ORIGINAL_MESSAGE:
Example given in the docs:
@Bean
@ServiceActivator(inputChannel = "pubsubInputChannel")
public MessageHandler messageReceiver() {
    return message -> {
        LOGGER.info("Message arrived! Payload: " + new String((byte[]) message.getPayload()));
        BasicAcknowledgeablePubsubMessage originalMessage =
              message.getHeaders().get(GcpPubSubHeaders.ORIGINAL_MESSAGE, BasicAcknowledgeablePubsubMessage.class);
        originalMessage.ack();
    };
}

Does this not work?

Thank you, perfect 👌
It is stated in the getting started guide: https://spring.io/guides/gs/messaging-gcp-pubsub/

Thanks! I filed an issue under the quick start guide.


Yarn

Test yarn, and make sure it works.
THEN:  Update docs to have yarn examples.
THEN: Open discussion ticket on moving internal npm commands to yarn
https://code.facebook.com/posts/1840075619545360
https://yarnpkg.com/en/compare

@juddey - you and several others suggested this :)

Well, it was more that it might be worth taking a look at and evaluating, but hey, if it works, lets jump in with both 🐾 .

looks like we're awaiting an update for yarn before we go forward any further.  Yarn latest is kinda broken.

Yeah, currently the yarn run command is borked, but that should be fixed in the next version.

Version 16.1 is out for yarn


chore: Create a garbage collection reconciler for v1alpha5.Machine

Fixes #
Description

Create a garbage collection reconciler for v1alpha5.Machine
Re-organizes sub-reconciler testing into separate files

How was this change tested?
make presubmit
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

21 of 23 (91.3%)  changed or added relevant lines in 4 files are covered.
12 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.06%) to 78.485%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/controllers/machine/initialization.go
1
3
33.33%

Files with Coverage Reduction
New Missed Lines
%

pkg/controllers/deprovisioning/controller.go
3
76.54%

pkg/controllers/deprovisioning/events/events.go
9
80.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.06%

Covered Lines:
6278

Relevant Lines:
7999

💛  - Coveralls


Fix #1855

Makes the deserialization inside readArray @trusted if it isn't already. In deserialization there shouldn't happen any infinite recursive loops so I think this won't be a problem.

Thanks! I really hope that this system can be cleaned up w.r.t. recursive structures and attribute inference one day... it got a bit messy trying to support @safe.


[1899] Allow editing of study mode

Context
Course study mode
Changes proposed in this pull request
Allow editing of study mode
Guidance to review
i.e. /organisations/1JA/2019/courses/2ZRL/full-part-time
After

 Edit 'Full, part time or both?' (MCFE)

@dankmitchell I spoke with Matt L and @davidgisbey about whether we should be allowing users to switch to "Full time or part time", as we don't yet have the capacity to allow them to pick which locations are full time and which are part time.
If they pick that option we wanted to direct them to support.


Fix: Error moving other tools' handles with bidirectional active

Good catch! This prevents that strange behaviour when the tool is active and you interact with other handles, thanks!

:tada: This PR is included in version 4.14.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


include gtags vim in Dotfiles vim

[x] create plugin dir of vim
[x] have reference to plugin somehow

Done


debugger can't find dotnet executable to start

Environment data
dotnet --info output:
Microsoft .NET Core Shared Framework Host

  Version  : 2.0.3
  Build    : a9190d4a75f4a982ae4b4fa8d1a24526566c69df

VS Code version: Version 1.18.1 (1.18.1)
C# Extension version: 1.13.1
Steps to reproduce
Use osx
Install VSC, extension.
Download dotnet core form site and use the install to setup.
Expected  behavior
Run the debugger
Actual behavior
having the same issue:
Displays the error after trying to debug.
launch: program '/usr/local/share/dotnet' does not exist.

I can run dotnet from internal terminal or OSX terminal. It also works if I assigned the debugger to an existing process, but doesn't work debugging a new one.

I am having what I think is a similar problem

I can dotnet run the program, I cannot debug it.

@MBLHarrison your issue is NOT similar. Please open a new issue.

@sourcesoft it looks like your problem is that the debugger things the path to the dotnet executable is /usr/local/share/dotnet, but on OSX in my experience it is actually /usr/local/share/dotnet/dotnet (note the extra `\dotnet'). So the debugger is pointing at a directory instead of a file.
If you look at your PATH, is /usr/local/share in it ahead of /usr/local/share/dotnet?

I just ran where dotnet in terminal:
/usr/local/bin/dotnet

So it actually runs the one in local/bin. And I also have /usr/local/share/dotnet folder with with a binary dotnet just like you said that I can also run like ./dotnet in there.

In the error I mentioned launch: program '/usr/local/share/dotnet' does not exist. it seems like it uses share folder path but I'm not sure why

Is /usr/local/share on your PATH?
The debugger does its owns processing of PATH to find dotnet, and I am guessing you are running into a bug in our parsing of it.

Yes I have it

And also the one in /usr/local/bin is just a symlink to the right path of dotnet

Okay, good. I should be able to repro this then. I will try to get the debugger fixed up for our next release. In the mean time, if you can swap the order of /usr/local/share and /usr/local/bin I think this will go away.

Oh it got worse. I just tested Visual Studio for Mac and it's getting me the same error text in a new solution/project, so it's not specific to VSC.

Both VS for Mac and VS Code use the same underlying debugger. So that makes sense.

I just swapped the order it got fixed in both of them. Thank you!


Pact should not allow the usage of REPL natives in module code

Pact legacy was disallowing by default REPL natives in module code.
Preventing developers testing modules that finally aren't deployable on chain.
Pact legacy was providing the convenient API (env-enable-repl-natives ) to however enable them for special cases.
Pact core always allows the usage of REPL natives in module code. IMO, it should not.
Since Pact core enables writing top-level functions, I'm not sure that (env-enable-repl-natives ) is still useful.
I suggest the following:

Always disable REPL natives in module context
But enable REPL natives in top-level functions (and ofc in top-level directly)
Do not implement (env-enable-repl-natives ) in Pact core

Since Pact core enables writing top-level functions, I'm not sure that (env-enable-repl-natives ) is still useful.

Top-level functions are repl-only. This is purely for convenience. I can disable them in .pact files in the future. They should only be enabled in .repl files.

Always disable REPL natives in module context

I'm ok with making it opt-in like prod. I've already used repl natives numerous times to debug contracts within pact-5 so I don't think I want to remove this capability.
I think we can implement env-enable-repl-natives with relative ease. I was already doing so in a separate branch, but work got halted for more important issues. We'll look into it again

This was completed by #114 in March.


v-analyzer

I'm not sure how to integrate language servers. But V recently deprecated there old one (which doesn't really work) and they have a new one called v-analyzer https://github.com/v-analyzer/v-analyzer
I have no idea on how to integrate it without using Mason.
Thanks!

Hello! See https://github.com/williamboman/mason.nvim/issues/1440


io_uring doesn't work in WSL2

Windows Version
Microsoft Windows [Version 10.0.22621.1848]
WSL Version
<IP_ADDRESS>
Are you using WSL 1 or WSL 2?

[X] WSL 2
[ ] WSL 1

Kernel Version
<IP_ADDRESS>
Distro Version
Ubuntu 22.04
Other Software
No response
Repro Steps
Install liburing on WSL2:
sudo apt install liburing-dev
Build & run the following C++ program and observe that sqeIn is null in WSL2 but valid in say a VirtualBox or a cloud-hosted Linux VPS running the exact same version of Ubuntu (22.04 in my case):
#include <string.h>
#include <liburing.h>
#include <netinet/in.h>

int main()
{
	io_uring ring;
	int result = io_uring_queue_init(32, &ring, 0);
	if (result != 0)
	{
		printf("error in io_uring_queue_init: %d, %s\n", result, strerror(-result));
		return 1;
	}

	io_uring_sqe* sqeIn = io_uring_get_sqe(&ring);
	if (!sqeIn)
	{
		printf("error in io_uring_get_sqe\n");
		return 1;
	}

	printf("io_uring verified\n");
	return 0;
}

Expected Behavior
"io_uring verified" should be printed to the console and sqeIn shouldn't be null.
Actual Behavior
"error in io_uring_get_sqe" is printed to the console and sqeIn is null.
Diagnostic Logs
No response

@GlassBeaver try this: https://github.com/ton-blockchain/ton/issues/431#issuecomment-1511391789
Sample run on WSL2:
tux@dragon:~/test3$ g++ test.cpp -luring
tux@dragon:~/test3$ ./a.out
io_uring verified

Fixed it. The issue was that I'd installed io_uring via apt. To make it work, I had to remove it and build from source:
sudo apt remove liburing-dev
sudo apt remove liburing2
git clone https://github.com/axboe/liburing.git
cd liburing
make && sudo make install

This works for me.
WSL2, <IP_ADDRESS>-microsoft-standard-WSL2


llama3.1-8b-it 模型加载报错

同样的模型 vllm/sglang/tgi3 都是ok的
17:06:53.763 Traceback (most recent call last):
17:06:53.763   File "/usr/local/conda/lib/python3.9/runpy.py", line 197, in _run_module_as_main
17:06:53.764     return _run_code(code, main_globals, None,
17:06:53.764   File "/usr/local/conda/lib/python3.9/runpy.py", line 87, in _run_code
17:06:53.764     exec(code, run_globals)
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/zhilight/server/openai/entrypoints/api_server.py", line 168, in <module>
17:06:53.764     engine = AsyncLLMEngine.from_engine_args(engine_args)
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/zhilight/server/openai/engine/async_llm_engine.py", line 161, in from_engine_args
17:06:53.764     engine = cls(engine_config,
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/zhilight/server/openai/engine/async_llm_engine.py", line 143, in __init__
17:06:53.764     self.engine = self._engine_class(engine_config = engine_config, log_stats = log_stats)
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/zhilight/server/openai/engine/llm_engine.py", line 39, in __init__
17:06:53.764     self._instance = LLaMA(
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/zhilight/llama.py", line 133, in __init__
17:06:53.764     self._init_tokenizer(vocab_path, tokenizer)
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/zhilight/llama.py", line 158, in _init_tokenizer
17:06:53.764     self._tokenizer.eos_token_id = self._config.get("eos_token_id", None) or 2
17:06:53.764   File "/usr/local/conda/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1077, in __setattr__
17:06:53.764     raise ValueError(f"Cannot set a non-string value as the {key}")
17:06:53.764 ValueError: Cannot set a non-string value as the eos_token

#20  this pr fix it


added text about certificates

almost a copy paste from Spring 2022 page. Maybe the credit can also be mentioned earlier in the page?

Thank you!

Even though it's a bit late now, but I feel like there should be more of a "make sure that your supervisor/study program accept it", since we can't do it for you (except e.g. Aalto)

Moving to an issue: #28
(this is very good suggestion but comments on merged PRs have high risk to get forgotten)


Why enforcing consistent reads?

Would be great to have an option to enable/disable consistent reads when triggering the queries.
Not a bug but it is for sure an improvement specially if you are searching a big area and want to save some read units. We can soon open a PR with this... I hope you consider.

BTW, this is the line you guys are enforcing this and overwriting what is on our client instance:
https://github.com/awslabs/dynamodb-geo/blob/1d0881295142a9bb15415817c644b104657666e5/src/com/amazonaws/geo/dynamodb/internal/DynamoDBManager.java#L92


Command bar opens / gets cloned twice, never re-enables chat when closed

What are the steps that lead to the bug occurring? Or, how frequently does the bug occur?
100% chance when opening command bar

What would be the normally expected or desired result?
It doesn't open twice

What is the actual result (the bug)?
It gets cloned twice to your player gui, thus adding TONS of friction to the process of executing commands + cannot open chat anymore

What is the impact of this bug?
Very high

(Any additional context or information eg. in the form of error messages or screenshots.)

There's both of em in my player gui

Video of the bug:
https://streamable.com/r067zd
Recorded on fresh empty baseplate, assuming this is a Roblox update breaking something

Should add, no visible error messages on client nor server

I thought it was something in my game messing with it. I tried opening it on a flat baseplate with adoni's only, but the same issue recurred.
o:

Should be fixed in new servers.


[FLINK-1730]Persist operator on Data Sets

This PR introduces a persist operation on DataSet which allows persisting the data set in memory, allowing for direct access if this data set is operated on again and again.
The idea behind the implementation is this:

A PersistOperator extending a SingleInputUdfOperator for common api and Java API.
A Persist driver strategy which allows the Job Graph to generate a PersistNode, which just uses a NoOpDriver to forward results from input to output.
RegularPactTask determines whether it is a Persist task and accordingly uses a SpillingResettableMutableObjectIterator to read the input and persist them.
To make the results truly persistent, the MemorySegments must not be freed when the Task ends. To this end, I have created a DummyPersistInvokable which does nothing. It just prevents freeing of memory.
All persisted memory segments are cleared out when the MemoryManager is shutting down. There is a possibility of writing some kind of Cache clearing strategy here.

For testing the functionality, I have written a test PersistITCase which generates 100 random Long values inside a Map function and persists the output. Then, triggering the execution twice must provide the same results.

Hi @sachingoel0101,
I think there was no consensus in the discussion (FLINK-1730) for how to implement this feature. @StephanEwen pointed that persisting data sets should happen on the network stack level, i.e., not on the operator level. In fact, this feature was considered and while redesigning and implementing the network stack by @uce. By doing it on the network stack level, persisted data sets can also be used for failure recovery or used across different jobs within a session.
This PR is basically ignoring the previous discussion and features of the network stack.
If you think that persisting on an operator level has advantages over the network stack, it would be good to share your thoughts with the community.
Unless there are good reasons to do persisting on operator level, I would not merge this PR.

Hey @fhueske , thanks for your comments.
I was not aware this was intended to allow for recovery on failed jobs.
For reusing among different jobs in the same session, I don't see how this doesn't solve the issue. If the Memory manager is alive, the results will be there for any job to use.
For a true across-job sharing, one possible feature would be to add a method for initialization from the environment as getPersistedSource(String) which would access results from a persisted data set from some entirely independent job.
Further, this kind of makes sense on an operator level. User should have to ability to explicitly persist a data set in memory, which calls for providing a function call. I was only drawing the analogy from spark's api. I have no idea how they internally implement this, but if an API function is to be provided, it can only be done in two ways. Either return a new Operator, as a transformation on the original data set, or just by returning the same data set [like withBroadcastSet does]. The former seemed easier to work with, because it doesn't interfere with the existing mechanisms.
I have implemented no new internal functionality, but only used the existing system. I would've loved more discussion on this but frankly, once I started going through the internal mechanisms, it seemed like a pretty trivial thing to implement. Of course that was when I wasn't aware it was intended to be used for recovery.
If there is some work on persisting intermediate results for recovery, the same mechanism can be used for a persist operation, in which case this work is anyways moot. But there has to be an API call to allow users to explicitly cache results in memory. This is a major problem I'm facing in implementing a randomized splitting algorithm.

You are certainly right that there should be an API call to explicitly persist data in memory (or transparently on disk if memory is short) and later access this data (within the same or another job). However, this feature can be implemented in different ways, for example using the network stack or on the operator level. Even if one implementation looks straight-forward, it can have severe limitations and implications on the behavior of the system. That is why such features should be discussed before taking action even if it looks like an easily doable thing.
Doing it on an operator level has several shortcomings:

persisted data sets cannot be used for recovery. If done on the network stack level, the same code can be basically used for both.
data cannot (easily) be shared across jobs. Operators are expected to return their memory when a job is done otherwise this will be a memory leak. There is no way to free memory if the job is finished and did not do it.

@uce, @StephanEwen You are more familiar with this feature. Did I miss something?

I don't think this really works. It violates so many assumptions, like the fact that memory is available after a job ends. The accounting for that depends on task slots - a free slot must have the necessary memory.
When you call collect twice, the whole program get re-executed and some parts are simply discarded for the sake of using the persisted data. That makes sure the results are the same, but does not safe any work. For that, you need backtracking through the graph to check whether results are already available.
Which brings us to the point:
Making the system's level of data streams aware of this (both on TaskManager and JobManager side) solves two issues with one approach: Caching for reuse in incrementally constructed programs, and caching for recovery. Both cases simply need persistent streams and backtracking in the execution graph.
Sorry that you made this effort and it cannot be merged now, but I do not understand why you keep ignoring the discussions, inputs, and the request to plan and describe such highly involved features before writing them.

I apologize. Like I said before, the way I've implemented this, it seemed pretty trivial. Foremost, I needed this for something else and decided to take a shot at it. Missed the finer details, going by your comments.
You're right though. Back-tracking the execution graph will most certainly be a better solution.
@StephanEwen , am I correct in assuming this functionality will be added after (or along with) Job Session management?
[As for the memory issue though, if data is to be persisted across jobs, we'll always have to keep the memory segments allocated even after the job finishes. This is why I defined a separate owner, which can be later garbage collected through a cache freeing mechanism.]
As far as merging it is considered, I wasn't sure myself from the beginning. I had actually written an email with a link to my branch for opinions, but ended up opening a PR instead of sending that. So no worries. I'm closing this.

Hi @sachingoel0101, session management is a prerequisite for sharing persisted data sets across jobs. However, persisting and reusing data sets is not part of PR #858.

@sachingoel0101 No need to apologize. It is just that you probably worked for quite a bit on this, and it is sad that this work is lost.
@mxm is merging the session management now, from there on we can go for proper caching/backtracking of intermediate results and solve this in a powerful way!

Great! One step closer. Would love to see this feature soon. :)

@sachingoel0101 I've opened a pull request some time ago to backtrack intermediate results on the network layer and then "backtrack" them during scheduling time: #640. At that time, I realized that this doesn't make much sense without a proper session management.

Aha. I had only searched Jira for any existing work on this. Anyways, I'm assuming you'll be rebasing the same PR after session management is in. I'm quite interested in this; already subscribed to your PR. :')


explorer getTransferByUser isn't working for testnet

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behavior:

Go to '...'

https://testnet.amarok.connextscan.io/address/0xfaa336d2aefdb7fc55ef4e3b14547d407223392f
And this address already has some transfers, ex:
https://testnet.amarok.connextscan.io/tx/0xd9851bf61eaf01e03facaf85042a5d2cc1d5887e3781855fe5b1cc3f54bdd76f
Additional context
Add any other context about the problem here.

@sanchaymittal This seems fixed i think?

i believe the same error happens in mainnet?
https://amarok.connextscan.io/address/0xd4dfee43d46a66471a988f3785106ea87c8f9f25
xcalled for example in:
https://etherscan.io/tx/0x1e95c54134910cb44b6f59b77547d25662f4398deea22ea0774e360322680efd/advanced


CUDA out of memory related to data parallel

I have a simple question related to the data parallel and your code.
I ran provided  train_lm.sh scripts. https://github.com/ethnhe/FFB6D/blob/master/ffb6d/train_lm.sh
When I ran the script, I got out of memory.
My main problem comes from the 0 GPU get more memories than others.
Q1. Have you experienced similar issues?? or this phenomenon is natural??
I observed that the number of workers affected 0 GPU usage.
Can you explain where the 147MiB is allocated in your code??
https://github.com/ethnhe/FFB6D/blob/3579a69df27451f74f3abc06964bba1bc7d40605/ffb6d/train_lm.py#L571
I think normalSpeed.depth_normal uses memory and the amount of memory depends on the size of workers. Does it right? Do you have a solution to make this memory allocates distributed??
https://github.com/ethnhe/FFB6D/blob/3579a69df27451f74f3abc06964bba1bc7d40605/ffb6d/datasets/linemod/linemod_dataset.py#L252
Current my GPU is TITAN Xp.

I solved the problem using some tricks.
If you can share some comments related to your experiences, It would be really helpful.
https://github.com/ethnhe/FFB6D/blob/3579a69df27451f74f3abc06964bba1bc7d40605/ffb6d/train_lm.py#L109
I changed CUDA_VISIBLE_DEVICES and others
os.environ['CUDA_VISIBLE_DEVICES'] = str(args.local_rank)
torch.cuda.set_device(0)
device = torch.device('cuda:{}'.format(0))
model = torch.nn.parallel.DistributedDataParallel(
model, device_ids=[0], output_device=0,
find_unused_parameters=True)

hey @taeyeop-lee  i just tried changing the lines like you did. I get the error that model is not defined for this
model = torch.nn.parallel.DistributedDataParallel(
model, device_ids=[0], output_device=0,
find_unused_parameters=True)


New 3.1.0 issue -- jsdoc failing:

assets/javascripts/actions/CommentActions.js
   5:2  error  JSDoc syntax error  valid-jsdoc
  19:2  error  JSDoc syntax error  valid-jsdoc
  29:2  error  JSDoc syntax error  valid-jsdoc
  39:2  error  JSDoc syntax error  valid-jsdoc

Line 5 is the first comment line.
class CommentActions {
  /**
   * Fetch comments from server.
   *
   * @param {String} url
   * @param {Boolean} displaySpinner
   * @return undefined
   */
  fetchComments(url, displaySpinner) {
    this.dispatch(displaySpinner);
    CommentsManager.fetchComments(url)
      .then((comments) => this.actions.updateComments(comments),
            (errorMessage) => this.actions.updateCommentsError(errorMessage));
  }

It's probably because your JSDoc comments are invalid. It's returns, not return.

@returns also fails.
You can try this out here.
https://github.com/justin808/react-webpack-rails-tutorial/pull/33
cd client
bin/lint

Yep, it was as I suspected. The errors are correct, your JSDoc is invalid. The valid JSDoc return annotation is @returns {type} not @return type