[Feature Request] Wormhole signature delete timer

Hi,
All wormholes have a maximum life time, it would be great if when a wormhole type sig is pasted into a map, it would automaticaly create a deletion timer (default 24hrs). Also would be great to be able to customize this timer.
Cheers

No, it wouldn't. Wormholes can remain unspawned if you warp to them, the only mechanism that should delete a signature is lazy delete.


⚠🚀 Deployments - change from VM to Blob storage

Cc: @adamcogan @Aibono1225 @bradystroud @JackDevAU

We need to decommission the VM we are deploying to and as a result switch to Azure Storage (blob storage)
### Tasks
- [ ] https://github.com/SSWConsulting/SSW.Rules/issues/1304
- [ ] https://github.com/SSWConsulting/SSW.Rules/issues/1301
- [ ] https://github.com/SSWConsulting/SSW.Rules/issues/1302
- [x] Change GitHub Action on Content repo to trigger this new action
- [ ] https://github.com/SSWConsulting/SSW.Rules/issues/1303

Duplicate - https://github.com/SSWConsulting/SSW.Rules/issues/1135
Thanks!

all done


Create ChangeColor.html

Your pull request needs further work because of the following reason(s):

Your filename must be PriyamAgrawal.html
Your file does not clearly show a link to your github profile.

Once you have resolved the issues, comment on this request to be re-judged
This is an automated response, beep boop.
if you think there is a mistake, contact a maintainer

5d285e77ae07f66f007a2d8ce033dc0d3c106f18

This looks like its a submission to week2, but it's in the wrong folder, it looks good otherwise.
Change the folder from Friends to Games/PriyamAgrawal/, create the other required files (main.js, main.css) and i'll be happy to merge.


Does not work with PS Nova Cart on 3.68

You said 15 days ago in #14 that you would reopen the case if your next version didn't fix the issue, will the Alpha version you released 6 days doesn't work, still can't use patches with physical cards.

The issue of #14 was that it softlocked on 3.68. Your issue is different. Please check your tai config.txt.

Well in my issue (#9), you linked #7 and #14 was linked in that last one, so I assumed that everything was part of the same issue. Anyway, I showed you screenshots of my config file, you know everything was well configured.

Oh yeah, alright. Well, did you do that eboot test?

@dots-tb
Ya, I did do the test, but you closed the topic before I could post the picture. Here, I did the test again again: https://s8.postimg.cc/8s2fwsqid/2018-07-30-211050.jpg

The config and plugins are in ur0:tai
The game updates and DLC are in ux0 in their appropriate folders
The games themselves are physical

I'm asking if the file that you are trying to patch actually exists within app (the one on the cart) or patch (maybe on the cart if its writable or on ux0).

I would assume yes because if I install PSNOVA as VPK, update the game with other VPK files and then replace the NOVA_FileList_Vita_patch.cpk inside the app folder for the translated one, then it works.

Please, when you can, check if the file is in the patch folder.

Official 1.05 update
https://s8.postimg.cc/4e35c67hh/2018-07-31-005312.jpg
Translation patch
https://s8.postimg.cc/pnqrn0vhx/2018-07-31-005410.jpg

Can you try renaming the original file in the official patch folder and seeing if that crashes the game?

https://s8.postimg.cc/f7ankt6t1/2018-07-31-014045.jpg

@dots-tb
Hi, just to let you know, I hooked a 2nd PSTV that I have that's on firmware 3.60, did the same setup and it works, so it's most likely a firmware issue.

https://github.com/dots-tb/rePatch-reDux0/wiki/Debug-Info-for-Bugs-and-Modding @the-importer I've added a page for getting log info. Please check it out.

@dots-tb
Me again, I was googling to see if there was a rePatch version for the translation of I Am Setsuna since I have a physical card for the game as well and found one. It was originally made for the old MaiDump version of the game, but someone made a rePatch version:
https://www.reddit.com/r/VitaPiracy/comments/8ct8px/i_am_setsuna_english_repatch_for_nonpdrm_repacked/
This actually works with my firmware 3.68, only different between this patch and PSNOVA/PSO2 is that it's not simply a CPK file, there's more to it.

@dots-tb
Sorry, but I'll need more info in order to setup ShipLog. It says "Install ShipLog.skprx in taiHEN config.txt"
Where's this ShipLog.skprx file?
Does taiHen = tai or is it somerthing different?

Yeah, I finally added the bug tag. Just let me know if you are able to getting logging information out of it.

@dots-tb
It says to type this command on my PC "nc -l -p 3333"
Tried in a CMD prompt, doesn't recognized the program.

If you don't want to use net, you can just create ux0:dump and then open the app and DUMP LOG BUFFER TO DISK and it will create logger.txt in that folder. "nc" is netcat, there are some windows version compiled on google or if you are on linux you can install it through your package manager software.

@dots-tb
Is this any good?
logger.txt

Can you launch a working game and then the non-working game? Or did you do that?

@dots-tb
I followed the instructions which didn't say anything about running games. You want me to run PSNOVA and then I Am Setsuna?

Well the instructions are just for setting up the logger. I would like that, yes.

OK, so I delete the logger file, booted PSNOVA and then I Am Setsuna. Here's the new file, but it doesn't look like it recorded any of the games I booted.
logger.txt

Are you using the debug version of rePatch on the wiki page? It should log every file call seen by rePatch.

No to both questions. In case it wasn't clear, I don't know anything about what you're making me do. I have no issues following instructions when it comes to computer stuff, but if those said instructions aren't clear or are lacking certain steps, then it's wasting both of our time. Is there anything else I need to know before doing this test again? Anything that wasn't in those instructions you've sent me?

@the-importer you need to replace regular repatch with debug repatch linked on wiki dots made.
that means this build: https://transfer.sh/7cJYQ/repatch.skprx

@dots-tb @TheRadziu
OK, this log file makes more sense: logger.txt

Both the PSO2 Vita English patch and the PS Nova English patch work on 3.68. They've both been tested by multiple people and confirmed working.

@RikuNoctis
Are you using a physical card or a rip?

@the-importer In my case, both of them.
The NoNpDrm version on my OLED Vita. The cart version on my slim Vita (I don't have SD2Vita on this one).

@Riku-Noctis
Well it fails to load on both my Vita-1000 and PSTV on 3.68, but loads properly on my PSTV that's on 3.60. My repatch setup is correct since I can load the translation of I am Setsuna.

I'm still on 3.65 personally, but people with both the cart and rip have confirmed it for me.

@Aida-Enna
I guess I can update my PSTV that's on 3.60 to 3.65 and see if that still works, that would help pin-point the issue.

@dots-tb
After reading that some people made this game work with h-encore and 3.67/3.68, I decided to see if my computer troubleshooting skills could solve this puzzle and indeed to were able to. I knew that there was   nothing wrong with the patches or the way I setup everything and if the firmware wasn't the issue, then what else was different with my PSTV and Vita on firmware 3.68 and my old one with firmware 3.60? Well, the memory card.
You see, I tried the patch on my 2nd PSTV using the internal 1GB of internal memory which was freshly formatted. On the other hand, on my main PSTV and my Vita 1000, I just swap my 32GB memory card between the 2 systems. That being said, I have a 16GB memory card which I stopped using, but still had some stuff on it. I inserted into my 2nd PSTV (3.60), copied the needed file from the internal storage and tried PS NOVA, the patch didn't work. So I formatted this card, had the system automatically copied to files from the internal storage to the 16GB memory card and tried PS NOVA again, it worked!!!
Now I have yet to format my 32GB card since I'll need to back a few things up first, but I'm 90% certain that this will work. That being said, I have no clue with I need to do this in the first place. It's not like rePatch wasn't working at all, the patch for I am Setsuna was working. My theories:
1- A pure coincidence that 2 memory cards had that problem (I don't personally believe in coincidences when it comes to computers)
2- A save data, patch or app on the memory cards was interfering with the patch for PS NOVA and PSO2
3- Perhaps the fact that the memory cards were linked to 2 accounts using Simple Account Switcher (the most likely culprit) was causing the incompatibility
I'll do some tests on the 16GB memory card and report back.

Nice to hear you got it working! And when in doubt, blame SilicaAndPina (Author of Simple Account Switcher).

@dots-tb @Aida-Enna
Well, everything is working fine now. That being said, I still do not know what caused the issue in the first place on both my memory cards. I've restored all of my save data, patches and DLC, h-encore is running fine, I've installed the 4 homebrews that I need are installed: VitaShell, Adrenaline, Simple Account Switcher & VitaRW and everything in my ur0 folder in both my Vita and PSTV remain the same (config.txt, all of the suprx and skprx files).
Only thing I can think of would be homebrews that I had installed, stopped using and removed, but some traced remained (Vitamin, MaiDump, etc..). So if this happening to you, I would suggests backing up your save data using QCMA, formatting your memory card, restore your data install the essentials (h-encore, VitaShell, Adrenaline) and see if the problems persists. If everything is working, then install other homebrews gradually and test. If one of your homebrews brakes rePatch for any game, then there's your culprit.
Sorry dots-tb if this has cause any inconveniences, never figured I'd see the days where I would need to apply a PC solution for a game system. I guess it's true what they say, these game systems are nothing but underpowered personal computers these days.


Name confusion: bigip or f5bigip?

Describe what happened
Our docsbot reported this:

:link: /registry/packages/f5bigip/ → https://pypi.org/project/pulumi-bigip/ (HTTP_404)
:link: /registry/packages/f5bigip/ → https://www.nuget.org/packages/Pulumi.Bigip (HTTP_404)
:link: /registry/packages/f5bigip/ → https://github.com/pulumi/pulumi-bigip (HTTP_404)

Looking at the sources, the provider name is configured as:
https://github.com/pulumi/pulumi-f5bigip/blob/537c787ac42d77517f45e9a9968c8058965186f6/provider/resources.go#L91
But the repository is named pulumi-f5bigip, which doesn't follow the usual pattern of pulumi-<name>. I assume, the newly introduced docs generation uses the configured name from resources.go, which leads to wrong package names in docs/_index.md:
https://github.com/pulumi/pulumi-f5bigip/blob/537c787ac42d77517f45e9a9968c8058965186f6/docs/_index.md#L10-L14
While the correct package names are:

Nuget: Pulumi.F5BigIp
NPM: @pulumi/f5bigip
Pypi: pulumi-f5bigip

Sample program
N/A
Log output
N/A
Affected Resource(s)
No response
Output of pulumi about
N/A
Additional context
No response
Contributing
Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

FYI the registry docs publishing process still has some hardcoded assumptions that the <name> of the provider also repeats in the repo name as pulumi-<name>. For instance, for community packages, the (short) name of the provider is extracted with regex like this in the GHA workflow:
https://github.com/pulumi/registry/blob/9232607d217762538c4ef4e58f8de672f02faf0e/.github/workflows/generate-package-metadata.yml#L40

Yikes. @ringods Do you know if there is a issue in the registry for this?

There doesn't seem to be one.

I've taken a look at this yesterday, and the issue is as follows:

info.Name is used to fetch information about the upstream provider, specifically the final part of the repo name after the last dash ("short name"). Example: if the upstream repo is terraform-provider-unicorn and our name is pulumi-unicorn then there's no issue either way. In this case, the upstream provider is bigip but ours is f5bigip.
The code for extracting the names in the bridge relied on info.Name because the above ^^ assumption wasn't clear, and because most of our providers are straight 1:1 matches wrt naming.
-> I think if we adjust the bridge code to get the short name similarly to that regex then this is a simple fix - all we need to do is extract the short name from something that specifically relies on the pulumi repo name.

As an aside, I also discovered that upstreamRepoPath is completely useless here, as it is a go module path and the only two valid settings for this info field are ./upstream (where we use a submodule in the provider) and unset (where the bridge sees an empty string and creates a go module path based on info.Name).


Can't add_child

I create hierarchy with:

parent

child1

In child1 had this script:
extends Node

func _ready():
    var parent      = get_node("..")
    var nodeTest    = Node.new()
    nodeTest.set_name("nodeTest")
    parent.add_child(nodeTest)
    print(parent.get_name(), " child = ", parent.get_child_count())
    var i
    for i in range(parent.get_child_count()):
        print(parent.get_child(i).get_name())

When run output showed:
parent child = 1
child1

It not add child in to parent node.

when you are in "_ready" the children are locked (because children are being iterated to give you that notification).
try doing
parent.call_deferred("add_child",nodeTest) instead

Ok, I understood.
Actually my work is load scene and at it into the parent node, and it had a script with function _ready, it must consider about call orders with other nodes.
Thanks reduz.

Can we put this in the documentation somewhere? I was following https://github.com/okamstudio/godot/wiki/tutorial_scripting_2, didn't know this, and was wondering why I was unable to add nodes and crashing the game.

It would be great if somehow some day the Godot engine didn't lock the list but instead made a duplicate for the iteration it is doing internally, so that when our scripts try to do things in _ready() they just work.

And what would happen if you deleted child nodes in _ready() while Godot is busy iterating over the list copy?

The list is only locked for parent nodes, the current node and it's
children can use add_child fine
On Thu, Jan 12, 2017 at 7:53 AM, pallavnawani<EMAIL_ADDRESS>wrote:

And what would happen if you deleted child nodes in _ready() while Godot
is busy iterating over the list copy?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/issues/1629#issuecomment-272133605,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF-Z2x8iwMiDzHJLniESlyER3lfe-PVNks5rRgYigaJpZM4D6bHQ
.

For me, the engine said ... System.NullReferenceException ... Node.cs:386 @ void Godot.Node.AddChild(Godot.Node , Boolean )() in my C# project. It ended up being that I was not calling base._Ready() and that I had to forgotten to assign a value to a timer in _Ready (timerProp=GetNode("Timer") as Timer;)

We need some kind of error notification for this. Just lost an hour trying to figure out what's going on. I was able to compile & run the code fine even though child node was never actually added.


Add instructions for usage as npm package

PTAL, thanks!

So now we have a single NPM package generated (thank you for that 👍) and deployed to the NPM registry, but how is this gonna be maintained?
(released with every new version of the dictation_support library)
I would assume that it was manually generated, because I don't see an action/workflow step added, or anything that ties the NPM package release into the release pipeline.
For future maintenance and support, I would love to see this as the result of a new version release, automatically created and uploaded.

We can already see how we have a 1.0.2 released as an NPM package, but no released 1.0.2 version of the library on the releases page. I think these two should be tightly linked together and not handled manually.

Also, small thing but probably useful. The released NPM package doesn't have a the GitHub repository set, so its not pointing to this repository on the NPM site, and not listed here as an existing package.
Thanks again for taking the time to improve the library's usability!

I have added the repository link to the package.json so it properly links back from npmjs.com.
I have never set up automated npm publishing, but it surely is feasible with an automated github workflow action and a generated access token. I don't have the cycles to take care of that at the moment and given that I/we don't expect this package to change on a regular basis, I think manually running npm publish is a reasonable alternative for now.
@rpriewasser-sps could you also create an account on npmjs.com and then I can add you as owner there as well so you can also publish it yourself. Feel free to add anyone else required.
Since I have switched teams internally, I would expect you (and your team) to handle long term maintenance of this package.

Thanks @alex292, I´ve created an account on npmjs: rpriewasser-sps

I've added you as maintainer to the npm package


Change schema

Description

Steps to Reproduce

start id flow

Expected Result
On page "your data is being verified", no other app (cryptoX, testnet) appears.
Actual Result
On page "your data is being verified", the user is asked if he wants to open the id object in e.g. cryptoX or testnet.
Versions

1.0.22
mainnet

Verified on testnet CCD v1.2.2 (64) with installed testnet, mainnet and cryptox wallet in newest versions.


about multi GPU training error

Hi, while I training this model on multi GPUs some error happened
My code is
`from torch.utils.data.distributed import DistributedSampler
local_rank = torch.distributed.get_rank()
torch.cuda.set_device(local_rank)
model = create_model(args,load_head=True)
model = torch.nn.DataParallel(model,device_ids=device_ids)
model.to(device)
for i, (inputData,target) in pbar:
# for i, (inputData, target) in enumerate(train_loader):
inputData = inputData.to(device)
target = target.to(device)  # (batch,3,num_classes)
# target = target.max(dim=1)[0]
with autocast():  # mixed precision
output = model(inputData).float()  # sigmoid will be done in loss !
# print("target shape = ",target)
# print("output shape = ",output)
loss = criterion(output, target)`
But error happened:
RuntimeError: Assertion THCTensor_(checkGPU)(state, 3, input, output, weight)' failed. Some of weight/gradient/input tensors are located on different GPUs. Please move them to a single one. at /opt/conda/conda-bld/pytorch_1603728993639/work/aten/src/THCUNN/generic/SpatialDepthwiseConvolution.cu:16
`
When I changed
self.filt = filt[None, None, :, :].repeat((self.channels, 1, 1, 1)).cuda().half()           to  self.filt = filt[None, None, :, :].repeat((self.channels, 1, 1, 1)).cuda(device=int(os.environ.get('RANK', 0))).half()
as you mentioned in TResNet
But the same error happened
RuntimeError: attribute lookup is not defined on python value of type '_Environ':   File "/home/kpl/code/multilabel/ML_Decoder/src_files/models/tresnet/layers/anti_aliasing.py", line 35         filt = filt / torch.sum(filt)         # self.filt = filt[None, None, :, :].repeat((self.channels, 1, 1, 1)).cuda().half()         self.filt = filt[None, None, :, :].repeat((self.channels, 1, 1, 1)).cuda(device=int(os.environ.get('RANK', 0))).half()                                                                                ~~~~~~~~~~~~~~ <--- HERE
you said you added an option --remove_aa_jit.
run with it, it should be ok.
But I don't find  --remove_aa_jit
can you give some suggestions?
Thanks very much.

i made some changes to the anti_aliasing.py file, try again and let me know if the issue persists.
p.s.
DDP (DistributedDataParallel) is more recommend than DataParallel for multi-GPU usage in pytorch.

i made some changes to the anti_aliasing.py file, try again and let me know if the issue persists.
p.s. DDP (DistributedDataParallel) is more recommend than DataParallel for multi-GPU usage in pytorch.

OHHHHHHHHHHHHHHHH，great job，it is useful. Now I can use multi GPUs.

i made some changes to the anti_aliasing.py file, try again and let me know if the issue persists.
p.s. DDP (DistributedDataParallel) is more recommend than DataParallel for multi-GPU usage in pytorch.

by the way,can you share the idea of ACC computing,I can get some meaning ，but can't get macro significance.
Besides, I think the code should be like this
ap1 = np.zeros((preds.shape[0]))
# compute average precision for each class
for k in range(preds.shape[0]):
# sort scores
scores = preds[k:, ]
targets = targs[k:, ]
# compute average precision
ap[k] = average_precision(scores, targets)
return 100 * ap.mean()
because, if in this batch if no class one,this batch target is [0,0,0,0,,,,,,] , and if preds is  [0,0,0,0,,,,,,]. This result means the predict is right but the acc is 0.

i am not sure i understood your messege.
what is "ACC computing" ?
'because, if in this batch no class one,this batch target is [0,0,0,0,,,,,,] , and if preds is [0,0,0,0,,,,,,]. This result means the prediction is right but the acc is 0. "
in a typical multi-label dataset (COCO, OpenImages), i think that you are guaranteed that each image has at least one positive class.

i am not sure i understood your messege. what is "ACC computing" ?
'because, if in this batch no class one,this batch target is [0,0,0,0,,,,,,] , and if preds is [0,0,0,0,,,,,,]. This result means the prediction is right but the acc is 0. " in a typical multi-label dataset (COCO, OpenImages), i think that you are guaranteed that each image has at least one positive class.

the  "ACC computing" is " mAP_score = validate_multi(train_loader, model, ema)"
Yes, image has at least one positive class. But your code is for k in range(preds.shape[1]) not for k in range(preds.shape[0])   the dim[0] is batch, dim [1] is classes  so.
 for k in range(preds.shape[1])
    scores = preds[:, k]
    targets = targs[:, k]

means [batch,1] not [1,classes]. theredor i think the code should be  for k in range(preds.shape[0])

i think it's ok.
it's a standard mAP calculation code. you aggregate the results from all the validation data (preds.shape[0]== num validation images, not batch), calculate mAP per class, and then average.
see similar code in:
https://github.com/Megvii-Nanjing/ML-GCN/blob/master/util.py#L220


API function call getMessage() for dialog boxes cuts off any content over 200 characters

Type
Bug
Operating System
Windows
OS Version
Windows 11
Bug description
The API does not return the full dialog for text over 200 characters. Any characters after 200 are just cut off from the result.
An example is per the screenshot, the actual output of getMessage or related functions (like MessageNode.getvalue) will instead return:

Ceril Carnillean|My son is a good boy, really, but how could he give his dinner to Scruffy without having the servants test it for poison first? How? How could he be so thoughtless and careless? He kn

Since the name of the NPC is returned with the result, the amount cut off at the end depends on the length of the NPC name.
Screenshots or videos

RuneLite version
1.10.19
2.6.10
Logs
No response

This is a Jagex limitation and can't be fixed by us. If you need to read a longer message, either read it from the dialog interface or maybe just do startsWith() for matching.

Gotcha, thanks for the quick response. I'll find a workaround somehow!


Beautify timeline

The timeline currently shows an excerpt of changes to resource provides but do not show all the details. I think it might make sense to limit the timeline overview to just the amount of changes comparable to a git commit (e.g. lines deleted, lines added) and show the full diff when opening up more information.

Agree, not enough time to rework as described, however have now collapsed the namespaces, can be expanded as needed + full diff available through button.  Looking forward to v2 to give a better summary of the changes in the overview 👍


Backport 4236

Read the original file for comparing/detecting newline (#4236)
Closes #4097
The code base diverged enough where a simple cherry-pick wasn't possible, but implementing the logic to read the input file when NewlineStyle::Auto was simple enough to implement.
New tests were added to hopefully prevent regressions.

I want to note that the original PR removes code that this PR does not since they take slightly different approaches to how apply_newline_style is called / updated. I haven't looked too deeply into it, but I wonder if we can remove similar code after making these changes.
I was also wondering if we had any tests in test/source that used crlf newlines. I was thinking it might be a good idea to add at least one system test for this backport to ensure that we maintain the newline style of the original input text regardless of the platform when newline_style=Auto

because of updates made to apply_newline_style, there will be some conflicts minor conflicts with #5418 depending on which backport PR gets merged first.


DBZ-1394 Add SMT to extract inner value field

https://issues.jboss.org/browse/DBZ-1394
Accessing inner values of records (inside before/after, depending of the event type) in other transforms (e.g. to change the records key) is currently not possible without applying event flattening with io.debezium.transforms.ExtractNewRecordState.
It would therefore be great if Debezium comes with a transform that allows extracting a field from the inner value (from before on delete operation, otherwise from after without the need to loose information through flattening.

Hi @bfncs, thanks a lot!

without the need to loose information through flattening.

I'm trying to wrap my head around this part.
IIUC, this extract-field SMT will propagate only a single field, so doesn't this amount to the same level of discarding information? I think the SMT makes sense as a usability improvement (instead of "Flatten" + "ExtractField" only a single one is needed), but the outcome seems to be the same as what you already can achieve today. Or am I missing anything?

Hello! I was thinking if maybe you could achieve that with the native SMTs, like ExtractField or maybe the ReplaceField?

@renatomefi, ExtractField doesn't work with nested structures afaik (i.e. you cannot extract "after.id"), so there's a usability improvement here.

@gunnarmorling @renatomefi @bfncs I understand the motivation behind this PR but if possible I'd propose a different approach I am cherishing for a quite time in my mind. The extraction and manipulation SMTs a are generally limited as I see them. Maybe we could make a step ahead and provide an integration with an expression language that would allow you to manipulate the message in larger terms.
Let's look at EL 3.0

it is standardad
standalone (no deps)
and lightweight (76 KB) - https://mvnrepository.com/artifact/javax.el/javax.el-api/3.0.1-b06

It can be integrated with Kafka Connect Struct using https://docs.oracle.com/javaee/7/api/index.html?javax/el/ELProcessor.html with a pretty low effort.
The net result would be that wou could easily make expressions like e.g. Integer.valueOf(in.before.id) . The SMT would allow to change both key and value depending on passed expressions.
WDYT?

Thanks for your helpful comments and questions. I'll try to elaborate a bit on the use case where I came up with the SMT.
I need to stream a MySQL table where the semantic key, that has to be used as message key for correct partitioning, is not the primary key of the table. Also in downstream processing I need the full before and after values and metadata, so I can't discard any data, only augment it.
Currently I'm achieving this by combining the ExtractConnectField SMT from the PR with ValueToKey.
"transforms": "ExtractConnectField,ValueToKey"
"transforms.ExtractConnectField.type": "io.debezium.transforms.ExtractConnectField"
"transforms.ExtractConnectField.field": "myDesiredKeyField"
"transforms.ValueToKey.type": "org.apache.kafka.connect.transforms.ValueToKey"
"transforms.ValueToKey.fields": "myDesiredKeyField"

So if you have this record before applying the transformation:
{
  "key": {
    "id": 1
  },
  "value": {
    "before": {
      "id": 1,
      "myDesiredKeyField": "123"
    },
    "after": {
      "id": 1,
      "myDesiredKeyField": "124"
    },
    "op": "u",
    "source": {}
  }
}

You get this record after applying it:
{
  "key": {
    "myDesiredKeyField": "124"
  },
  "value": {
    "before": {
      "id": 1,
      "myDesiredKeyField": "123"
    },
    "after": {
      "id": 1,
      "myDesiredKeyField": "124"
    },
    "myDesiredKeyField": "124",
    "op": "u",
    "source": {}
  }
}

IIUC, this extract-field SMT will propagate only a single field, so doesn't this amount to the same level of discarding information? I think the SMT makes sense as a usability improvement (instead of "Flatten" + "ExtractField" only a single one is needed), but the outcome seems to be the same as what you already can achieve today. Or am I missing anything?

It is not actually the same, @gunnarmorling, not only a single field is propagated but the complete original message (with before, after, op, source) with the extracted field added.

Hello! I was thinking if maybe you could achieve that with the native SMTs, like ExtractField or maybe the ReplaceField?

I can't use ExtractField, @renatomefi, because I want to extract the value from after if it is a create, update or read operation but from before if it is a delete operation, similar to how ExtractNewRecordState already works.
I think it's a beautiful idea to integrate a lightweight manipulation expression language, @jpechane, but at least for my use case (and similar ones I can imagine), most manipulations can already be done with existing SMTs. It is currently only problematic to extract the correct field depending on the operation to enable the usage of other, well known SMTs in the first place, and this one could potentially fix it.
Maybe I'm also not seeing a possibility to achieve the same thing. I was thinking about implementing the transformation in a Kafka Streams application, but since I have no use for the non-transformed data anyway, that seemed a bit heavy-headed, so it would be great to have an upstream SMT for that. Let me know what you're thinking.

@jpechane, it's an interesting idea, but opens up a whole other can of worms. On the size, you only gave the API, you'd also have to add an actual EL implementation. But the much larger implication is that of performance; SMTs by themselves already add their share to processing time, and EL execution will come on top. Not saying it might not be useful for certain cases, but I think for simple ones like extracting a field it adds prohibitively much overhead.
I'd still like to hear back from @bfncs on the "without the need to loose information through flattening." part, because it's still not quite clear to me whether this new PR solely is a usability improvement or whether it actually is meant to enable something you cannot do as of today.

I'd still like to hear back from @bfncs on the "without the need to loose information through flattening." part, because it's still not quite clear to me whether this new PR solely is a usability improvement or whether it actually is meant to enable something you cannot do as of today.

I hope my last response made that clear, but just to be sure: for all what I know this is not possible without custom SMTs as of today, because you either can't access the correct field in before or after depending on the operation or you loose metadata through flattening.

@gunnarmorling I am sorry, I forgot amout impl - https://mvnrepository.com/artifact/org.glassfish/javax.el-impl/3.0.1.b08-redhat-1 it is 171 KB which is still nice.
I understand the performance implications but I can image that for more complex scenarios this could be good start for easy config and when performance is not enough the user can write the Java code as an optimization.

Thanks for clarifying, @bfncs, I understand the use case now. Being able to customize what Debezium emits as a primary key is a feature request we've seen before actually (DBZ-1015).
We could integrate the SMT as a stop-gap solution,but ultimately this is something which should be made customizeable in connectors themselves by means of a new connector option for specifying the column(s) forming the message key. Would you perhaps be interested in giving this approach a try?

Thanks for refering to DBZ-1015, I haven't seen it before but it definitely makes sense to integrate it at this level to provide a better usability.
On the other hand I don't think adding this SMT would be limited to a stop-gap solution, because it enables using data from Debezium with all other existing SMTs. It's hard to come up with a genuine use case for this that is not already possible apart from changing the key, so it might still be the best choice not to add this to the codebase.
I'll certainly have a look at how to implement configurable key columns, but I suspect this to be quite a complicated change, so no promises I'll come up with a solution soon.

I'll certainly have a look at how to implement configurable key columns, but I suspect this to be quite a complicated change

That's great. I don't think it would be too complex actually. Essentially, have a look where there PK column(s) are passed to the TableBuilder class; this would "just" have to be driven by a config option instead of the actual PK columns. Of course, devil, details, and so on :)

Caused by: org.postgresql.util.PSQLException: ERROR: could not load library "/home/postgres/pgsql/lib/decoderbufs.so": /lib/liblwgeom.so.1: undefined symbol: DirectFunctionCall1
when  i  run
./connect-standalone.sh ../config/connect-standalone.properties ../config/postgres.properties
[root@shuais-super-big--vm bin]# ldd /home/postgres/pgsql/lib/decoderbufs.so 	linux-vdso.so.1 =>  (0x00007ffd2d4c4000) 	libprotobuf-c.so.1 => /usr/local/lib/libprotobuf-c.so.1 (0x00007feb327f9000) 	liblwgeom.so.1 => /lib/liblwgeom.so.1 (0x00007feb3258f000) 	libc.so.6 => /lib64/libc.so.6 (0x00007feb321c2000) 	libgeos_c.so.1 => /usr/local/lib/libgeos_c.so.1 (0x00007feb31f90000) 	libproj.so.0 => /lib64/libproj.so.0 (0x00007feb31d3d000) 	/lib64/ld-linux-x86-64.so.2 (0x000055f9c0c33000) 	libgeos-3.6.3.so => /usr/local/lib/libgeos-3.6.3.so (0x00007feb3198f000) 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007feb31688000) 	libm.so.6 => /lib64/libm.so.6 (0x00007feb31385000) 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007feb3116f000) [root@shuais-super-big--vm bin]# ldd iblwgeom.so.1 ldd: ./iblwgeom.so.1: No such file or directory [root@shuais-super-big--vm bin]# ldd /lib/liblwgeom.so.1 	linux-vdso.so.1 =>  (0x00007ffca75d4000) 	libgeos_c.so.1 => /usr/local/lib/libgeos_c.so.1 (0x00007fc92e97c000) 	libproj.so.0 => /lib64/libproj.so.0 (0x00007fc92e72a000) 	libc.so.6 => /lib64/libc.so.6 (0x00007fc92e35c000) 	libgeos-3.6.3.so => /usr/local/lib/libgeos-3.6.3.so (0x00007fc92dfae000) 	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fc92dca7000) 	libm.so.6 => /lib64/libm.so.6 (0x00007fc92d9a4000) 	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc92d78e000) 	/lib64/ld-linux-x86-64.so.2 (0x000055e752c5c000)

@shuaixiaozhou Hi, is the coment relevant to the PR?

Hey @bfncs, did you have a chance to look into that more generic implementation we discussed above?

@bfncs, I'm going to close this one, as I don't think in its current form the PR is in a good shape for inclusion into Debezium, and I'd recommend you maintain such SMT just locally for yourself.
Specifically, I don't think the duplication between the "extracted" field at the top-level of the structure and its original location within the "after" part is desirable (it's different from the behavior of the upstream ExtractField SMT which replaces the entire key/value with the extracted field).
For addressing your use case, a PR for customizing the message key (DBZ-1015) would be very welcomed. Thanks!

Hey @gunnarmorling,
sorry, I didn't find any time to look into this due to holidays etc. Thanks for taking this into consideration, I also think solving the underlying issue by making the message key configurable would be much better. I'll definitely have a look if I can contribute in the closer future.

Hey @bfncs, no worries, and thanks for getting back to this. Your contributions will be definitely welcomed. Note that for the specific issue of making the message key configurable there's an active PR (https://github.com/debezium/debezium/pull/1009) in the workings by @willome.


"Kernel Died With Exit Code 1" whenever I try running jupyter notebook in a custom env in VS code.

I have tried updating and force updating dependencies but it still doesn't work.
Here is the full message Kernel died with exit code 1. Traceback (most recent call last):

Thanks for filing this issue & I'm sorry you're running into this issue, however to help you resolve this issue I'll need you to fill in the issue template. Please could you provide the following missing information.
Here's the information we require
Environment data

VS Code version: XXX
Jupyter Extension version (available under the Extensions sidebar): XXX
Python Extension version (available under the Extensions sidebar): XXX
OS (Windows | Mac | Linux distro) and version: XXX
Python and/or Anaconda version: XXX
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): XXX
Jupyter server running: Local | Remote | N/A

Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

XXX

Logs

Output for Jupyter in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Jupyter)

XXX

Closing due to inactivity


📄🚀 – Relax constraints as much as possible

Describe the feature you want and how it meets your needs or solves a problem

As a producer of data, I want to easily provide my data in TIDES format so that I can take advantage of tools designed to work with the spec. I want to use the spec to tell vendors what the minimum data requirements are, and how they can validate their output. I am willing to forgo immediate utility of the data in favor of more tools that help me understand and enhance my validated data.
Describe the solution you'd like

I want to relax as many required constraints as possible, throughout the spec.

vehicle_locations

[x] trip_id_performed, #71
[ ] stop_sequence
[ ] date
[ ] maybe even location_ping_id, which could be replaced by vehicle_id, timestamp and conditionally, device_id.

fare_transactions

[ ] fare_capped

passenger_events

[ ] trip_id_performed
[ ] stop_sequence

TODO: summary tables, supporting tables

Some enum fields might be better as strings to support agency specific values, or enum options should be expanded.

[x] Convert route_type to a string, #85

Describe alternatives you've considered

TODO
Additional context and sample data

Here's my understanding of the discussion from the 2022-11-02 contributor meeting:
There are two key issues:

how to define needs related to TIDES in contracts
how to guarantee functionality, the ability to do useful things with the TIDES data

The proposed resolution is to eliminate the required constraint from as many fields as possible, making the spec more flexible, but reducing the guarantees for what can be done directly with fully valid data.

By converting as many fields as possible to optional we lose some functionality guarantees, but don't need to worry about specifying multiple tiers of spec adherence (Gold, Silver, Bronze!), or predicting which fields would be required for which purposes.
The remaining required fields really do represent the minimum requirements, and so the spec can be used as-is for contracts, or agencies can upgrade individual fields to required. Either way, the resulting data will validate, and a vendor can verify they've met the contract terms by passing validation.
Data that pass the bare minimum spec may not be very useful on their own. Additional tools, not specified in the spec, and with their own data requirements may be needed to polish these rough datasets into analytic jewels.
The widespread use of a minimal spec will generate demand for such polishing tools and there are likely to be multiple options with different features and data requirements.
The use of a minimal spec also facilitates the development of QA/QC tools based on the spec, instead of stipulating that data in the spec already pass quality control checks.
The datapackage.json file that should accompany data files allows for arbitrary metadata that can be used by producers to indicate the status of the data (raw, cleaned, whatever)
The datapackage.json file should not be used to describe the properties of the data, and if it is, consuming tools should not trust it. For example, do not claim that there are no duplicate keys, or that there are no null values. It is the responsibility of the consuming tool to verify the data meet their needs and raise the appropriate errors or warnings as needed.

I'd like to move this issue forward so it can be resolved for V1:
vehicle_locations

Per Issue #112, agree that stop_sequence, renamed to scheduled_stop_sequence should not be required.  New trip_stop_sequence should also not be required
Agree that service_date should not be required
I think location_ping_id should be required, or do we also create similar alternate unique keys for fare_transactions.transaction_id and passenger_events.passenger_event_id (shouldn't these be parallel)

fare_transactions

Agree fare_capped should not be required (or is the rationale that boolean value should be required.  I'd prefer a default assumption if the value is not specified, in this case FALSE)

passenger_events

Agree trip_id_performed should not be required (similar to vehicle_locations), but if trips_performed table is populated it would be wise to link passenger_events via the trip_id_performed!

I don't see anything else on the Summary Tables or Support Tables.

Unless there is disagreement, I will implement per @jlstpaul 's comment on April 12th.

This PR removes requirement constraints for service_date, fare_capped, trip_id_performed, stop_sequence as follows:

vehicle_locations.stop_sequence requirement constraint removed
vehicle_locations.service_date requirement constraint removed
fare_transactions.fare_capped requirement constraint removed
passenger_events.trip_id_performed requirement constraint removed
passenger_events.trip_id_performed and  vehicle_locations.trip_id_performed Added recommendation for "If trips_performed table is used, this must be populated in order to link to it."

No Action

vehicle_locations.location_ping_id retained as required since that recommendation didn't seem firm.


Unlock User Page

Need to adjust spinner based on the response object that is coming back from the API.

This seems to be working at intended.


hf.GetExitError() and hf.GetExitCode() are unreachable

//hf.Log("Exit Error: " + hf.GetExitError())
//hf.Log("Exit Code: " + hf.GetExitCode())

💊 Capsule shutting down gracefully ...
👋🤗 have a nice day
2022/09/18 16:26:28 wasm error: unreachable
wasm stack trace:
        .runtime.runtimePanic(i32,i32)
        .runtime.lookupPanic()
        .allocateBuffer(i32) i32
😡[execVoidFunction] wasm error: unreachable
wasm stack trace:
        .runtime.runtimePanic(i32,i32)
        .runtime.lookupPanic()
        .allocateBuffer(i32) i32
 (recovered by wazero)
wasm stack trace:
        env.hostGetExitError(i32,i32)
        .OnExit()
🔴 Error wasm error: unreachable
wasm stack trace:
        .runtime.runtimePanic(i32,i32)
        .runtime.lookupPanic()
        .allocateBuffer(i32) i32
 (recovered by wazero)
wasm stack trace:
        env.hostGetExitError(i32,i32)
        .OnExit()
💊 Capsule exiting

I removed these host functions in v0.3.4 🍋 [lemon]


PWM example for EDISON

Is this line ok?
PWM are on MRAA 21, 0, 14 and 20 why it is using 3? I get an error using 3.
I don't get error using 21, but not PWM output on PWM3, GP183...

Edison with which baseboard? With arduino board 3 is valid on default swizzler positions (I've never tried to support any other!)

So the example is intended to be used with the Arduino board, that's why it uses pin 3. I think we've clarified this, so closing. Feel free to comment if any further help is needed.


Setting cover art for work displays "Updating avatar" text

When setting the cover art for a work, the popup that appears says "updating avatar" rather than more appropriate text.

corrected in 1.0.0-alpha.10


Nma notifications

NMA (Notify My Android) notification support.

Had a minor comment and Travis found some style violations. For the rest looks good to go

Looks like Travis still reports a couple of style violations:
$ flake8 homeassistant --exclude bower_components,external
homeassistant/components/notify/nma.py:48:9: F401 'Request' imported but unused
homeassistant/components/notify/nma.py:59:40: E203 whitespace before ':'
homeassistant/components/notify/nma.py:75:9: F401 'Request' imported but unused
homeassistant/components/notify/nma.py:78:31: E203 whitespace before ':'
The command "flake8 homeassistant --exclude bower_components,external" exited with 1.

$ pylint homeassistant
************* Module homeassistant.components.notify.nma
W: 64,30: Use % formatting in logging functions but pass the % parameters as arguments (logging-format-interpolation)
W: 98,34: Use % formatting in logging functions but pass the % parameters as arguments (logging-format-interpolation)

Hmm, I seem to have a discrepancy between the local tests with pylint/flakes8 and the ones travis runs...

Travis installs the latest version of flake8 and pylint every time it runs the tests. Running pip3 install --upgrade flake8 pylint should fix it.

Coverage decreased (-0.58%) to 62.62% when pulling 14d0c1325c7691d12a72f07a52f7c85aeb748b77 on fabaff:nma into 362e1763979d39a0e0c07ce4dddf4d385f0045f4 on balloob:dev.


Add script to precompile Ops for RNN tutorials

The code seems fine. However, since a lot of people might be cloning this repo to get the presentation material/exercices, we should probably have a comment in there to explain the purpose of this file. I wouldn't want people to be confused by this.

I merge now to have the VM ready.


How to get Object from a ACAD_OC Dictionary

Hi
wonder if someone can help me with this
I need to get info about object in Dictionary
ACAD_OC
Test - 10.02.2020 - Copy = just a path to a file
32mm = Polyline
600L 480mm br = block
72 Fiber = Polylinee

Regards
Sigster

Hi @sigster
In the image you seem to be looking at the equivalent in ACadSharp CadDocument.RootDictionary, you may be able to access into the dictionary using CadDocument.RootDictionary["ACAD_OC"] which I'm guessing that it will be another dictionary that you can access into their entries and get the values.
Let me know if this helps.


Multiple Environment Support Fixed

Proposed change
When using a central portainer instance with multiple agents they were all showing up under a single device, fixes #23
Type of change

[X] Bugfix
[ ] New feature
[ ] Code quality improvements to existing code or addition of tests
[ ] Documentation

Additional information

Checklist

[X] The code change is tested and works locally.
[X] The code has been formatted using Black.
[ ] Tests have been added to verify that the new code works.
[X] Documentation added/updated if required.

Marking as unstale ;-)
@tomaae


Add rmg.co.uk to buyer email domains

Trello: https://trello.com/c/cK8XlvhY/454-add-additional-buyer-email-domains-for-buyer-accounts

👍


Difference btw decode_tacotron2.py vs tacotron2_inference.ipynb

The procedure of running inference of tacotron2 is not clear. It appears that running only "tacotron2_inference.ipynb" looks enough to me. What is "decode_tacotron2.py" for? Even running this code gives me errors as below.
2020-07-01 01:59:24.756772: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow d
evice (/job:localhost/replica:0/task:0/device:GPU:0 with 299 MB memory) -> physical GPU (device: 0, name:
 GeForce GTX 1080 Ti, pci bus id: 0000:08:00.0, compute capability: 6.1)
2020-07-01 01:59:43.496259: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcublas.so.10
2020-07-01 01:59:43.723366: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully
opened dynamic library libcudnn.so.7
2020-07-01 01:59:44.157435: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn han
dle: CUDNN_STATUS_INTERNAL_ERROR
2020-07-01 01:59:44.167314: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn han
dle: CUDNN_STATUS_INTERNAL_ERROR
2020-07-01 01:59:44.172894: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn han
dle: CUDNN_STATUS_INTERNAL_ERROR
2020-07-01 01:59:44.175197: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn han
dle: CUDNN_STATUS_INTERNAL_ERROR
Traceback (most recent call last):                                                             [105/1355]
  File "examples_models/tacotron2/decode_tacotron2.py", line 136, in <module>
    main()
  File "examples_models/tacotron2/decode_tacotron2.py", line 103, in main
    tacotron2._build()  # build model to be able load_weights.
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/models/tacotron2.py", line 677, in _build
    self(input_ids, input_lengths, speaker_ids, mel_outputs, mel_lengths, 10, training=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 968, $
n __call__
    outputs = self.call(cast_inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 580, in __$
all__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py", line 644, in _c$
ll
    return self._stateless_fn(*args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 2420, in __cal$
__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1665, in _filt$
red_call
    self.captured_inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 1746, in _call$
flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py", line 60, in quick_exe
cute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize,
 so try looking to see if a warning log message was printed above.
         [[node encoder/conv_batch_norm/tf_tacotron_conv_batch_norm/conv_._0/conv1d (defined at /usr/loca
l/lib/python3.6/dist-packages/tensorflow_tts/models/tacotron2.py:86) ]]
         [[decoder/while/body/_1/decoder_cell/assert_positive/assert_less/Assert/AssertGuard/pivot_f/_265
/_47]]
  (1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize,
 so try looking to see if a warning log message was printed above.
         [[node encoder/conv_batch_norm/tf_tacotron_conv_batch_norm/conv_._0/conv1d (defined at /usr/loca
l/lib/python3.6/dist-packages/tensorflow_tts/models/tacotron2.py:86) ]]
0 successful operations.
0 derived errors ignored. [Op:__inference_call_8385]

Function call stack:
call -> call

@jun-danieloh the error may because the GPU is used by another process i think. About the difference, tacotron2_inference demo the real inference on deployment including save it to pb file then load and do synthesis while decode_tacotron2 inference with input is the folder ids and it used in case you want inference all samples on the folders :D.

@dathudeptrai
Thanks for your quick response. Now I understand the diferences.
I have more questions. What's the output of "tacotron2_inference"? Is it mel? Then do I need to take this mel output to Griffin_lim inference code?

@jun-danieloh yeah, you can use that mel and pass to GL inference to check the output. You can also train vocoder like melgan or multiband melgan as vocoder :D. GL is bad quality and just use to check :D

@dathudeptrai Got it! Thanks for your clarity.
Last question. Do you know which tensorflow version is working properly with this repo, https://github.com/Rayhane-mamah/Tacotron-2? I saw your post in this repo so ask. I guess it would be 1.15? :)

:))) at the time i use that repo, i used tf 1.10

@dathudeptrai Thanks :)


Wrong Classification of all videos as Violent

After training the model on Hockey+Movie dataset the classifier works good on dataset videos but classifies all other random videos as violent. What do you think might be the issue ??

After training the model on Hockey+Movie dataset the classifier works good on dataset videos but classifies all other random videos as violent. What do you think might be the issue ??

Can you  share me other videos as violent?

This means you should enrich your dataset.  The reason that I only use the Hockey dataset is to compare the performance of the network with the previous works.  If you want to deploy it to the real world, you must collect as much data as you can.  This may be the aka problem of the artificial neural network...
Moreover, if you found the training loss can't get down to what you expected after you collecting many data, you can add more layer to gain the capability of the network.

thank you so much


fix: Fix update subscription method

Fixes #2119.

This PR contains a fix for the continuous build failure.


Incompatible ANSI escape sequence at the end

Prerequisites

[x] I have read and understand the CONTRIBUTING guide
[x] I looked for duplicate issues before submitting this one

Description
While tinkering around with Clink, the author (@chrisant996) of the revived project discovered a bug in the default theme of OMP3. This does not cause any functional issues, but renders the theme incompatible with Readline libraries on many platforms.
The details of the discovery are mentioned here chrisant996/clink#39, while explanations and reproduction steps are mentioned in chrisant996/clink#40.
Environment

Oh my Posh version: 3.61.1
Theme: Default
Operating System: any
Shell: any
Terminal: any

It causes a functional issue in Clink, and should cause exactly the same functional issue in Bash, since both use the GNU Readline library which is the component affected by the oh-my-posh issue.
The CSI K escape code near the end of the prompt string can, under certain circumstances, accidentally erase the input text from the screen. Readline doesn't expect anything else to be printing over its output, and for performance reasons it's important not to print over or erase Readline's output. But the CSI K escape code does.
The linked issue 40 has repro steps for one example scenario. Many other scenarios exist.

@chrisant996 The clear until eol escaped sequence is there because Powershell has a bug where it colors the line after the prompt when the prompt scrolls up when it's at the bottom of your console. So removing that will break Powershell, but it can be a Powershell only feature obviously.
About the speed, it's weird but people have been reporting it's faster than starship, much to my surprise, although I noticed Windows is the slowest of the bunch for some unknown reason.

We already discussed this in the past but what slows down the prompt on windows is the git segment.
Maybe we could check what's done differently in starship on this point?

the git segment

To be honest, we do A LOT more than others in regards to git functionality and I would like to keep it that way. I did create a workaround for the sys calls, which should have avoided that timeout as much as possible without having to fork go.

the git segment

To be honest, we do A LOT more than others in regards to git functionality and I would like to keep it that way. I did create a workaround for the sys calls, which should have avoided that timeout as much as possible without having to fork go.

True, I checked starship and it shows less informations(at least by default) than omp.

@lnu I'm going to create a POC to see if we can delegate the syscalls to a Rust lib. It's perfectly possible and it could solve the issue entirely. See this repo for example.

@chrisant996 The clear until eol escape sequence is there because Powershell has a bug where it colors the line after the prompt when the prompt scrolls up when it's at the bottom of your console. So removing that will break Powershell, but it can be a Powershell only feature obviously.

Ah, mutually exclusive issues -- the most fun of all problems. 😄
Maybe some kind of callout in the documentation could even be sufficient. As you say, some way to influence the behavior based on what shell is targeted.

About the speed, it's weird but people have been reporting it's faster than starship, much to my surprise, although I noticed Windows is noticeably slower for some unknown reason.

I've only just discovered and installed both. I'll try some comparisons and share what I learn. I might also take a couple minutes to see if the Visual Studio profiler reveals any significant time consumers.
For me, oh-my-posh takes around 2 to 3 seconds to run on Windows when using the default theme. I'll post more specific details when I get a chance.

Also, care to clarify point 2 more specifically?

Sure. It generates an almost 2 KB string for around 80 characters of text. It seemed there are many series of color escape codes back to back, often redundant, with no text in between.
Perhaps something about my environment hits a case not normally seen. When I get a chance I'll upload a file containing the full string, and maybe annotate redundant or superfluous sequences.

We already discussed this in the past but what slows down(mostly) the prompt on windows is the git segment.
Maybe we could check what's done differently in starship on this point?
And generally, syscall are really slow in windows compared to mac or linux.

If you can point me where to look in the oh-my-posh sources for the relevant git stuff I can compare it with what the Lua git prompt scripts for Clink do. Maybe that could reveal some interesting differences. The Lua scripts take maybe half a second for the git prompt info.
The oh-my-posh executable seems very large on Windows. It's over 8 MB, which suggests it may have a lot of framework code. Sometimes large frameworks hurt load and/or initialization performance. Just an observation; I don't have any data yet about where time is consumed.

True, I checked starship and it shows less informations(at least by default) than omp.

I think this misses the point. Starship is, for most purposes, not a ready-to-go prompt utility; it shows very few things by default and needs a configuration file. I have uploaded my starship.toml here for reference.

True, I checked starship and it shows less informations(at least by default) than omp.

I think this misses the point. Starship is, for most purposes, not a ready-to-go prompt utility; it shows very few things by default and needs a configuration file. I have uploaded my starship.toml here for reference.

I did not take the time to configure it extensively. Thanks for your config.

oh-my-posh takes around 2 to 3 seconds to run on Windows

@chrisant996 that's definitely not normal. It runs below a second on my machines (and also confirmed by others). The only time this occurred, it disappeared after a reboot of the user, and as far as I know hasn't returned just yet.
A lot has been said in this thread, @rashil2000's original issue will be tackled in #275 as I'm working on that right now. Redundant ANSI escape sequences have been taken out by @lnu, which hopefully makes things a bit more efficient. I'll create a POC to see of the syscalls on Windows can be bypassed, but no guarantees there until proven otherwise.

@rashil2000 once v3.65.0 is done building, can you validate this resolves your issue?

Unfortunately, I can still hear the terminal bell when oh-my-posh (v3.65.0) is injected in Clink. Same as before.

Wait, the bell? That wasn't mentioned before, right? This fix was for the clear EOL.

Ok, so, I just reread the entire thread in 39 and see the mention of the bell and indeed that's because of setting the title. Setting console_title for false will resolved that.

@rashil2000 can you confirm that the original issue as described 40 is now gone?

oh-my-posh takes around 2 to 3 seconds to run on Windows
@chrisant996 that's definitely not normal. It runs below a second on my machines (and also confirmed by others). The only time this occurred, it disappeared after a reboot of the user, and as far as I know hasn't returned just yet.

Stopwatch says it's around 1 second.  I guess it feels like 3 seconds because Lua scripts in Clink can do the same work (including git enhancements) in around 1/4 second, but that's to be expected since they don't have to load a large binary or initialize internal runtime frameworks, etc.

Redundant ANSI escape sequences have been taken out by @lnu, which hopefully makes things a bit more efficient.

Cool, sounds great.
It sounds like all of the issues are well understood, so at this point I think there's no need for me to collect/upload additional information.

oh-my-posh takes around 2 to 3 seconds to run on Windows
@chrisant996 that's definitely not normal. It runs below a second on my machines (and also confirmed by others). The only time this occurred, it disappeared after a reboot of the user, and as far as I know hasn't returned just yet.

Stopwatch says it's around 1 second. I guess it feels like 3 seconds because Lua scripts in Clink can do the same work (including git enhancements) in around 1/4 second, but that's to be expected since they don't have to load a large binary or initialize internal runtime frameworks, etc.

Redundant ANSI escape sequences have been taken out by @lnu, which hopefully makes things a bit more efficient.

Cool, sounds great.
It sounds like all of the issues are well understood, so at this point I think there's no need for me to collect/upload additional information.

@chrisant996 thanks for taking the time to assist on this one. It clarified a lot and made us improve things 🙏🏻

Setting console_title to false will resolve that.

How do I set this?

"console_title": false ìn the general settings of the json config file.

@rashil2000 can you confirm that the original issue as described 40 is now gone?

Yes, no error bell now!
Thanks!


How to get an ApplicationSet to spawn Applications in Argo CD?

I've been installing the ApplicationSet together with Argo CD 2.0.0 and 2.0.1 without any success in spawning applications from a any generators. Maybe I am missing something trivial, but how do you get the ApplicationSet into Argo CD?
I have tried adding an Application in Argo CD with a path to a folder in the git repo that contains a spec for an ApplicationSet, or several. I can see the argocd-applicationset-controller running as a replica set and stateful set.
This is what it looks like in Argo CD...

Hi @WillooWisp , I suggest double checking that the ApplicationSet controller is installed into the same namespace as the Argo CD you are connecting to.
Next, I suggest trying some of the examples, for example, the very simple list generator example.
You should just be able to do:
kubectl apply -f examples/list-generator/list-example.yaml -n argocd

And then within moments kubectl get applications should show the Application that was created, named engineering-dev-guestbook. If that doesn't work, it may be advantageous to check the ApplicationSet logs for any obvious red flags.

It is in the same namespace. So I guess the problem might be that there is no way of adding an ApplicationSet using Argo CD CLI or UI which is what I have tried.

Hi @WillooWisp, it depends on the policy flag you run applicationset-controller binary. You probably have the applicationset conroller deployment with this spec.
So the default policy flag is sync which will delete the application in case you delete the directory from git.
If you want to prevent it you can use --policy=create-update or --policy=create-only.
You can read it here .

Okay, but the template part in the application set, generating the application, supports argocd specific elements like plugin an sync options?

I must be doing something very wrong but I cannot get applications to spawn.
Now I run 0.1.0 version of applicationset, I've also tried the very latest.
I run...
kubectl apply -f examples/list-generator/list-example.yaml
applicationset.argoproj.io/guestbook created

No guestbook application gets added.
Logs from applicationset controller pod...
2021-06-08T10:55:34.641Z	INFO	setup	ApplicationSet controller using namespace 'argocd'	{"namespace": "argocd"}
2021-06-08T10:55:35.268Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-06-08T10:55:35.270Z	INFO	setup	Starting manager
2021-06-08T10:55:35.372Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
2021-06-08T10:55:35.372Z	INFO	controller-runtime.manager.controller.applicationset	Starting EventSource	{"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "source": "kind source: /, Kind="}
2021-06-08T10:55:35.474Z	INFO	controller-runtime.manager.controller.applicationset	Starting EventSource	{"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "source": "kind source: /, Kind="}
2021-06-08T10:55:35.474Z	INFO	controller-runtime.manager.controller.applicationset	Starting EventSource	{"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "source": "kind source: /, Kind="}
2021-06-08T10:55:35.576Z	INFO	controller-runtime.manager.controller.applicationset	Starting Controller	{"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet"}
2021-06-08T10:55:35.576Z	INFO	controller-runtime.manager.controller.applicationset	Starting workers	{"reconciler group": "argoproj.io", "reconciler kind": "ApplicationSet", "worker count": 1}

That worked so all applications sets must belong to the same namespace as argocd?

@WillooWisp Correct: in the same way that with Argo CD all Argo CD Applications need to be in the Argo CD namespace (eg argocd), all ApplicationSets need to be there as well.


error of same type of different paths and question of Cargo.toml git source rev

See this :
https://github.com/hyperium/hyper-tls/issues/4
@alexcrichton

Ah yes this is expected behavior, with different rev specifications those are two different sources. You'll want to use cargo update precise to lock the rev most likely rather than specify it in the manifest


Moving Forward

I know not technically an issue but I wanted to alert everyone following this project.
Even though it is still not very big, this does seem to be gaining support, where we even have a possible candidate for our icon! So, with that I plan on refining the current code base and making it more functional/readable.
Currently, it is a rough implementation of the interfaces stated in the wiki. In the coming weeks I plan on developing a few helper classes to aid in the design of these components and future ones.
I also realize I haven't been active due to work conditions, but with this new design, helping out should be quite a bit easier.
So look forward to these changes and changes that will occur on this project asking for help on things that may be easy to accomplish.
Thank you to everyone that has shown interest and have been giving their time! I think there is a lot of potential here and that with this we could build a new generation of applications written in web languages but targeted for native platforms. Thanks!
Justin

Cool, will love to see at least an API guideline to open the way to start working on other platforms besides windows.
And don’t worry it may take time to gain support, but when the project get to an usable state (even unstable) it will start gaining traction and we could publish on hacker news or reddit to try to get attention.

Wheres the PR ticket on the scrum board, I'll gladly promote this project.

I'll also gladly promote the project!

@jscherer92 More power your way. This is such an important and useful thing. Do you have plans for macOS? I think macOS support can boost the project quite a bit, I know lots of friends building small utilities and menubar apps for macOS (including myself). Quark can be a perfect fit there.
So, I don't know much Swift or Obj C, but I can gladly promote the project to my friends who know or help on the JS, Web, or Node APIs part if you wanna mimick them.
So... the more info we have, the more we can help! You're awesome :D

Thanks!
I am planning on getting the PoC of the windows platform back up and running, but I plan on going a different direction with the project, mainly with the language choice.
It may seem counter productive to do, but I wanted to use a lanaguage that will be a bit more inviting to use than C++ but still have the same performance characteristics.
I will be talking about this fact soon. Sadly, I have not had near enough time to commit to this project due to work and outside commitments, but I plan on getting back to it once things settle down.
I will do the write up soon on my thoughts and a possible PoC for MacOSX using this new method.

This is a great idea. A roadmap would be good and a basic API overview, to start.  Have you considered using Rust?

@jscherer92 thanks for updates in the Readme!

closing out as a good chunk of communication will be done in the README for now


Using the level builder to edit existing levels

Is it possible to use the level builder to edit existing levels? I tried opening the level builder and typing import level, but that didn't seem to work.
If that feature doesn't exist, it seems like it would be a useful thing to have. Right now, I'm basically stuck having to edit the JSON manually.

@pcottle but how to edit my own levels? I have built a custom level, exported that and saved into a gist. Is it possible to edit that level using the level builder?

There is the level edit command but I think it might only work for built-in levels with an ID.
I would perhaps just make a clone of the project locally and add your level as a real level, and then use the level edit command there

Hmm, did you mean build level instead of level edit? It seems to be broken at the moment, for example, when I enter command build level rebasing (rebasing is a valid level ID?), then I get an error in console Uncaught Error: need goal tree and solution 🤔

Sorry try edit level instead
On Wed, Aug 26, 2020 at 8:12 AM Janno Teelem<EMAIL_ADDRESS>wrote:

Hmm, did you mean build level instead of level edit? It seems to be
broken at the moment, for example, when I enter command build level
rebasing (rebasing is a valid level ID?), then I get an error in console Uncaught
Error: need goal tree and solution 🤔
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pcottle/learnGitBranching/issues/187#issuecomment-680905343,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAIVDHYPAKWJSZ5P2EBLHNLSCUJ6DANCNFSM4ARGGS2A
.
--
-Sent from Gmail Mobile

edit level doesn't work either: command not supported. Something seems to be broken, I'm trying to debug a little bit.

Ah yeah sorry again, I must be totally mis-remembering. It's been years since I've had to edit a level, I might have either had a debug-only way to load up existing levels or just been editing the JSON manually. That's probably the best going forward!
As a reminder you can export the current tree from the sandbox environment with debug_copyTree() in the console.
So really only the dialogs need to be edited by hand in the JSON


YAML format of rules get should match rules sync

It would be helpful if the output format of cortextool rules get <namespace> <group> matched the exact format expected to cortextool rules sync.  This would make it easier to get down all rules and commit them into Git, then sync them back again.  Right now I have to munge the YAML slightly to make it compatible to sync.
Right now the format is this:
$ cortextool rules get somenamespace anygroup
name: anygroup
rules:
    - alert: FrontEnd Prometheus
      expr: .......

Ideally it would be this:
namespace: somenamespace
groups:
    - name: anygroup
      rules:
        - alert: FrontEnd Prometheus
          expr: ........

A simper test:
$ cortextool rules print --disable-color > tmp.yml
$ cortextool rules sync tmp.yml
ERRO[0000] unable parse rules file                       error="yaml: unmarshal errors:\n  line 1: field group1 not found in type rules.RuleNamespace\n  line 13: field group2 not found in type rules.RuleNamespace" file=tmp.yml
cortextool: error: sync operation unsuccessful, unable to parse rules files: file read error, try --help

A simper test:
$ cortextool rules print --disable-color > tmp.yml
$ cortextool rules sync tmp.yml
ERRO[0000] unable parse rules file                       error="yaml: unmarshal errors:\n  line 1: field group1 not found in type rules.RuleNamespace\n  line 13: field group2 not found in type rules.RuleNamespace" file=tmp.yml
cortextool: error: sync operation unsuccessful, unable to parse rules files: file read error, try --help

To circumvent this annoyance I've written a small script to convert the content generated by coretextool rules print.
$ cortextool rules print --disable-color > tmp.yml
$ ./split.py tmp.yml
$ rules check --rule-dirs=rules && echo OK
OK

#!/usr/bin/env python3

import argparse
import os
import yaml

def _load_yaml(path):
    with open(path) as file:
        return yaml.load(file, Loader=yaml.FullLoader)

def _write_yaml(data, path):
    with open(path, "w") as file:
        yaml.dump(data, file, sort_keys=False, width=9999)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("namespaces_file", type=str)
    args = parser.parse_args()

    base_dir = os.path.dirname(os.path.abspath(args.namespaces_file))
    rules_dir = os.path.join(base_dir, "rules")
    os.makedirs(rules_dir)

    for namespace, groups in _load_yaml(args.namespaces_file).items():
        _write_yaml(
            data={"namespace": namespace, "groups": groups},
            path=os.path.join(rules_dir, f"{namespace}.yaml"),
        )

if __name__ == "__main__":
    main()


Condor is not able to access runtime scripts from WMAgent container

Impact of the bug
WMAgent
Describe the bug
In short, when running jobs from a container-based WMAgent, there is an inconsistent path for the runtime WMAgent code which crashes grid jobs right away.
The jobs are not even reaching the grid.  They fail directly into the agent with the following error: [1]. The most important line is:
                <s>Error from slot1_5<EMAIL_ADDRESS>SHADOW at 2001:1458:d00:39::100:49 failed to send file(s) to <[2620:6a:0:8421:f0:0:190:77]:41669>: |Error: reading from file /usr/local/lib/python3.8/site-packages/WMCore/WMRuntime/Unpacker.py: (errno 2) No such file or directory; STARTER failed to receive file(s) from <[2001:1458:d00:39::100:49]:4080></s>

Htcondor puts the job on hold immediately, and what we see is basically only the condor error logs and nothing related to the WMagent  jobs.
Here is all that is preserved in the job archive:
cmst1@vocms0290:Job_1 $ ll
total 20
-rw-r--r--. 1 cmst1 zh 8145 Mar 18 16:58 condor.3958.0.log
-rw-r--r--. 1 cmst1 zh 2732 Mar 18 16:53 job.pkl
-rw-r--r--. 1 cmst1 zh 4514 Mar 18 16:59 Report.0.pkl

One is for sure, the origin of the problem is not hidden behind the difference in the job working directory path set inside the container and the host:
/data/dockerMount/srv/wmagent/2.3.0/ vs. /data/srv/wmagent/2.3.0/

Because this  one  is already resolved with a symbolic link created at  the host:
cmst1@vocms0290:Job_1 $ ls -la /data/srv/wmagent
lrwxrwxrwx. 1 cmst1 zh 29 Mar 18 15:31 /data/srv/wmagent -> /data/dockerMount/srv/wmagent

The Unpacker script itself lives in the regular path for all WMcore modules (including Runtime), which is: WMCore/src/python/WMCore/, and it ends up in the container's WMA_DEPLOY_DIR itself which in our case is:
(WMAgent-2.3.0) [cmst1@vocms0290:current]$ echo $WMA_DEPLOY_DIR
/usr/local

And this is also quite visible from the error message itslef. The Unpacker script is searched for at:
/usr/local/lib/python3.8/site-packages/WMCore/WMRuntime/Unpacker.py

Which is inside the container and not part of the host mount point. This is a location completely invisible for HTCondor.
NOTE:
For the purpose of this investigation, I also ran this same test in a virtual environment, which lacks the extra encapsulation coming from the Docker container, so HTCondor and the Agent runtime environment should see all the host files without the extra bind mount complications, but the error still exists:

    <s>Transfer input files failure at access point vocms0290 while sending files to the execution point. Details: reading from file /data/WMAgent.venv3/srv/WMCore/WMCore/WMRuntime/Unpacker.py: (errno 2) No such file or directory</s>

Which makes me think here may be involved a wrongly set environment variable at the agent during the deployment  which should be referring to the correct path at runtime. WMCORE_ROOT is one that comes to mind, which is set with the manage script:   https://github.com/dmwm/CMSKubernetes/blob/192b3037b26d05d29ea668da80f93ce48303d013/docker/pypi/wmagent/bin/manage#L14-L22
# NOTE: The folowing environment variables, which are referenced from inside the WMCore code,
#       are exported by sourcing the $WMA_ENV_FILE at runtime, but not from the Dockerfile at build time
#
#       * $WMAGENTPY3_VERSION
#       * $WMAGENTPY3_ROOT
#       * $WMCORE_ROOT
#       * install
#       * config
#       * X509_*

It is located here:
(WMAgent-2.3.0) [cmst1@vocms0290:current]$ echo $WMA_ENV_FILE
/usr/local/deploy/env.sh

And its contents:
(WMAgent-2.3.0) [cmst1@vocms0290:current]$ cat /usr/local/deploy/env.sh
cd $WMA_ROOT_DIR/srv/wmagent/current

# exporting the variables to the enviroment
export WMAGENT_SECRETS_LOCATION=$WMA_ROOT_DIR/admin/wmagent/WMAgent.secrets
export X509_HOST_CERT=$WMA_ROOT_DIR/certs/servicecert.pem
export X509_HOST_KEY=$WMA_ROOT_DIR/certs/servicekey.pem
export X509_USER_CERT=$WMA_ROOT_DIR/certs/servicecert.pem
export X509_USER_KEY=$WMA_ROOT_DIR/certs/servicekey.pem
export X509_USER_PROXY=$WMA_ROOT_DIR/certs/myproxy.pem
export install=$WMA_INSTALL_DIR
export config=$WMA_CONFIG_DIR
export manage=$WMA_MANAGE_DIR/manage
export RUCIO_HOME=$WMA_CONFIG_DIR

alias condorq='condor_q -format "%i." ClusterID -format "%s " ProcId -format " %i " JobStatus  -format " %d " ServerTime-EnteredCurrentStatus -format "%s" UserLog -format " %s\n" DESIRED_Sites'
alias condor_overview='python ~/condor_overview.py'
export WMA_BUILD_ID=$(cat $WMA_ROOT_DIR/.dockerBuildId)
export WMCORE_ROOT=$WMA_DEPLOY_DIR
export WMAGENTPY3_ROOT=$WMA_INSTALL_DIR
export WMAGENTPY3_VERSION=$WMA_TAG
export CRYPTOGRAPHY_ALLOW_OPENSSL_102=true
export YUI_ROOT=/usr/local/yui/
export PATH=$WMA_INSTALL_DIR/bin${PATH:+:$PATH}
export PATH=$WMA_DEPLOY_DIR/bin${PATH:+:$PATH}

How to reproduce it
Build a WMAgent  docker container and run it.
Upon that  send a set a workflows targeting the relevant WMAgent.
Expected behavior
Condor to be able to create the sandbox for the Job regardless of the fact if it runs inside or outside the container.
Additional context and error message
[1]
https://cmsweb-testbed.cern.ch/wmstatsserver/data/jobdetail/tivanov_ReReco_RunBlockWhite_WMAgentDockerTest_v2_240318_144614_1700
Errors:

    NoJobReport
        NoJobReport (Exit Code: 99303)

            Could not find jobReport

            ========== condor.3958.0.err ==========

            ========== condor.3958.0.out ==========

            ========== condor.3958.0.log ==========
            Adding end of condor.3958.0.log to error message:

                1710777519
                <s>T1_US_FNAL,T2_CH_CERN</s>
                1
                1710777519
                2
                <s>ULOG_SHADOW_EXCEPTION</s>
                <s>2024-03-18T16:58:47.670</s>
                3958
                <s>T1_US_FNAL</s>
                0
                <s>JobAdInformationEvent</s>
                <r>2.663137000000000E+06</r>
                28
                0
                <r>0.0</r>
                <s>Error from slot1_5<EMAIL_ADDRESS>SHADOW at 2001:1458:d00:39::100:49 failed to send file(s) to <[2620:6a:0:8421:f0:0:190:77]:41669>: |Error: reading from file /usr/local/lib/python3.8/site-packages/WMCore/WMRuntime/Unpacker.py: (errno 2) No such file or directory; STARTER failed to receive file(s) from <[2001:1458:d00:39::100:49]:4080></s>
                <s>T1_US_FNAL,T2_CH_CERN</s>
            </c>
            <c>
                13
                <s>Transfer input files failure at access point vocms0290 while sending files to the execution point. Details: reading from file /usr/local/lib/python3.8/site-packages/WMCore/WMRuntime/Unpacker.py: (errno 2) No such file or directory</s>
                2
                0
                12
                <s>JobHeldEvent</s>
                0
                3958
                <s>2024-03-18T16:58:48.122</s>
            </c>
            <c>
                1710777528
                <s>T1_US_FNAL,T2_CH_CERN</s>
                1710777198
                12
                <s>T1_US_FNAL,T2_CH_CERN</s>
                1710777519
                5
                <s>ULOG_JOB_HELD</s>
                <s>2024-03-18T16:58:48.122</s>
                3958
                <s>T1_US_FNAL</s>
                0
                28
                0
                <s>JobAdInformationEvent</s>
                2
                13
                <s>Transfer input files failure at access point vocms0290 while sending files to the execution point. Details: reading from file /usr/local/lib/python3.8/site-packages/WMCore/WMRuntime/Unpacker.py: (errno 2) No such file or directory</s>
                1
            </c>

Hi @amaltaro  I actually did something more than just updating the issue's description. I've also put there all the information I have collected during the investigation I was already doing last week. You may  take a look for more details.

And here is the fix for this:
https://github.com/dmwm/CMSKubernetes/pull/1460/files

And with those changes we just made one step further to the final goal but we did not achieve it in full:
Here is the next  condor error we have stumbled on: [1] . from which the most important part is:
Error: reading from file /usr/local/etc/submit.sh: (errno 2) No such file or directory;

[1]
Could not find jobReport

========== condor.3961.0.err ==========

========== condor.3961.0.out ==========

========== condor.3961.0.log ==========
Adding end of condor.3961.0.log to error message:

    1712000144
    <s>T1_US_FNAL</s>
    11
    1712000144
    2
    <s>ULOG_SHADOW_EXCEPTION</s>
    <s>2024-04-01T21:35:57.701</s>
    3961
    <s>T1_US_FNAL</s>
    0
    <s>JobAdInformationEvent</s>
    <r>2.674767000000000E+06</r>
    28
    0
    <r>0.0</r>
    <s>Error from slot1_8<EMAIL_ADDRESS>SHADOW at 2001:1458:d00:39::100:49 failed to send file(s) to <[2620:6a:0:8421:f0:0:190:18]:42129>: |Error: reading from file /usr/local/etc/submit.sh: (errno 2) No such file or directory; STARTER failed to receive file(s) from <[2001:1458:d00:39::100:49]:4080></s>
    <s>T1_US_FNAL</s>
</c>
<c>
    13
    <s>Transfer input files failure at access point vocms0290 while sending files to the execution point. Details: reading from file /usr/local/etc/submit.sh: (errno 2) No such file or directory</s>
    2
    0
    12
    <s>JobHeldEvent</s>
    0
    3961
    <s>2024-04-01T21:36:07.986</s>
</c>
<c>
    1712000167
    <s>T1_US_FNAL</s>
    1711982430
    12
    <s>T1_US_FNAL</s>
    1712000144
    5
    <s>ULOG_JOB_HELD</s>
    <s>2024-04-01T21:36:07.986</s>
    3961
    <s>T1_US_FNAL</s>
    0
    28
    0
    <s>JobAdInformationEvent</s>
    2
    13
    <s>Transfer input files failure at access point vocms0290 while sending files to the execution point. Details: reading from file /usr/local/etc/submit.sh: (errno 2) No such file or directory</s>
    11
</c>

One more painful step further with this commit: CMSKubernetes#1460/commits: Change config.JobSubmitter.submitScript location
 Its purpose is to copy also the submit.py script together with the rest of the environment and to tweak the wmagent config in such a way that it would be placed in a location reachable by HTCondor

Unfortunately the small but significant success from few minutes ago leads to yet another stone to stumble on:
The workflow submitted was: SC_MultiPU_WMAgentDockerTest_v7_240418_124104_3709
The jobs again reached to condor and tried to run. But here are the errors observed this time: SC_MultiPU_WMAgentDockerTest_v7_240418_124104_3709/JobDetails
Could not find jobReport

========== condor.8218.0.err ==========
Adding end of condor.8218.0.err to error message:

GWMS_SINGULARITY_BIND_GPU_LIBS=1
CVMFS_REPOS_LIST=
APPTAINER_BIND=/etc/grid-security,/etc/hosts,/etc/localtime,/cvmfs
OMP_THREAD_LIMIT=4
GWMS_SINGULARITY_LIB_VERSION=2
GLITE_LOCATION=/usr
GLIDEIN_PROVIDES=singularity/unprivileged
GLIDEIN_LOCAL_TMP_DIR=/tmp/glide_cmsplt02_4JPANo
_CONDOR_JOB_IWD=/srv
SITE_NAME=CERN-PROD
TF_LOOP_PARALLEL_ITERATIONS=4
SRM_PATH=/opt/d-cache/srm
GWMS_SINGULARITY_OUTSIDE_PWD=/pool/condor/dir_3830787/glide_JEQAVM/execute/dir_12139
GWMS_SINGULARITY_IMAGE_RESTRICTIONS=cvmfs
CONDOR_PROCD_ADDRESS=/pool/condor/dir_3830787/glide_JEQAVM/log/procd_address
_=/usr/bin/printenv
DEBUG (/srv/.gwms-user-job-wrapper.sh) GWMS singularity wrapper, final setup.
DEBUG (/srv/.gwms-user-job-wrapper.sh) current directory at execution (/srv): total 5820
drwx------. 5 cmsplt02 zh    4096 Apr 19 21:20 .
drwxr-xr-x. 1 cmsplt02 zh      40 Apr 19 21:20 ..
-rw-r--r--. 1 cmsplt02 zh   14462 Apr 19 21:20 .LOG_.gwms-user-job-wrapper.sh.21.txt
drwx------. 2 cmsplt02 zh       6 Apr 19 21:20 .apptainer
-rwx------. 1 cmsplt02 zh      48 Apr 19 21:20 .chirp.config
drwxr-xr-x. 5 cmsplt02 zh      56 Apr 19 21:20 .gwms.d
drwxr-xr-x. 2 cmsplt02 zh      40 Apr 19 21:20 .gwms_aux
-rw-r--r--. 1 cmsplt02 zh    9645 Apr 19 21:20 .job.ad
-rw-r--r--. 1 cmsplt02 zh   17503 Apr 19 21:20 .machine.ad
-rw-r--r--. 1 cmsplt02 zh   12018 Apr 19 21:20 JobPackage.pkl
-rw-r--r--. 1 cmsplt02 zh    4938 Apr 19 21:20 Unpacker.py
-rw-r--r--. 1 cmsplt02 zh   19302 Apr 19 21:20 _condor_stderr
-rw-r--r--. 1 cmsplt02 zh     121 Apr 19 21:20 _condor_stdout
-rwxr-xr-x. 1 cmsplt02 zh    7502 Apr 19 21:20 condor_exec.exe
-rw-------. 1 cmsplt02 zh   17544 Apr 19 21:20 myproxy.pem
-rw-r--r--. 1 cmsplt02 zh 5828100 Apr 19 21:20 tivanov_SC_MultiPU_WMAgentDockerTest_v7_240418_124104_3709-Sandbox.tar.bz2
DEBUG (/srv/.gwms-user-job-wrapper.sh) GWMS singularity wrapper, job exec: /srv/condor_exec.exe tivanov_SC_MultiPU_WMAgentDockerTest_v7_240418_124104_3709-Sandbox.tar.bz2 1 0
DEBUG (/srv/.gwms-user-job-wrapper.sh) GWMS singularity wrapper, messages after this line are from the actual job ##################
Traceback (most recent call last):
  File "Startup.py", line 18, in <module>
    from Utils.Timers import CodeTimer
  File "/srv/job/Utils/Timers.py", line 86
    self.logger.info(f"{self.label} took {runtime} seconds to complete")
                                                                      ^
SyntaxError: invalid syntax
+ cp 'WMTaskSpace/Report*.pkl' ../
cp: cannot stat 'WMTaskSpace/Report*.pkl': No such file or directory
+ ls -l WMTaskSpace
ls: cannot access WMTaskSpace: No such file or directory
+ ls -l 'WMTaskSpace/*'
ls: cannot access WMTaskSpace/*: No such file or directory
+ set +x

The "syntax error" in line 86 is suspicious. I remember getting those when python2 was used rather than python3 for some tests.

Thanks @khurtado! that's  a good hint.
I wonder if it could be related to the problems with the singularity container. I do not think we use python2 anywhere any more. There were some issues with the ITB pool last 24 hours. Maybe I should try to resubmit in the GlobalPool. If we observe the same errors there we could attribute it for sure the job runtime configuration.

@todor-ivanov One explanation could be that submit.sh is getting executed rather than submit_py3.sh

hmm.... good point....

Thank you @khurtado it was a supper useful remark of yours !!!
And,

Here is the commit to fix that silly mistake of mine: #CMSKubernetes/1460/commits:Fixing wrong submitScript
Here are 3 other workflows for us to monitor for the outcome:

SC_MultiPU_WMAgentDockerTest_v8_240419_214153_2257
ReReco_Run2022C_LumiMask_WMAgentDockerTest_v8_240419_214144_7576
ReReco_RunBlockWhite_WMAgentDockerTest_v8_240419_214137_806

I can already see the workflows completing. I've sent the same set of workflows to both: the GlobalPool (through vocms0290)  and to ITB (through vocms0260) . They behave absolutely the same (not that we would expect anything else).
The suggestion that @khurtado  was the fix to the last Error. Thank you Kenyi, that was really right on the spot.
I am merging the CMSKuberenetes changes and finally closing this issue now.


Endless create/leave loop when TZ changed to a TZ run that is not enabled.

commit: feba50097719754e9b0ce67bf0ca2d4b3011fd8a
It will create game, go to the TZ that is not enabled on the TZ list. and show game completed once it took waypoint there then leave and do again till the hour is over. If i restart bot it will resume as normal and skip this TZ since its not enabled. The current TZ during this was stony field (this was not enabled, but the prior hours TZ was jail, which is enabled.)
+skip all runs and only do tz when available is enabled.
time=13:28:03 level=INFO msg="Running: TerrorZone Run: Stony Field, Jail Level 1, Jail Level 2, Jail Level 3"
time=13:28:03 level=DEBUG msg="Loading screen detected, waiting until loading screen is gone"
time=13:28:08 level=DEBUG msg="Load completed, continuing execution"
time=13:28:08 level=DEBUG msg="Checking for character corpse..."
time=13:28:08 level=DEBUG msg="Character corpse not found :D"
time=13:28:08 level=DEBUG msg="Checking for items to identify..."
time=13:28:08 level=DEBUG msg="No items to identify..."
time=13:28:08 level=DEBUG msg="Belt Stats Health: 8/8 healing, 4/4 mana, 4/4 rejuv."
time=13:28:08 level=DEBUG msg="Checking for items to stash..."
time=13:28:08 level=DEBUG msg="No items to stash..."
time=13:28:08 level=DEBUG msg="Checking for items to stash..."
time=13:28:08 level=DEBUG msg="No items to stash..."
time=13:28:08 level=DEBUG msg="TZ area skipped" area="Stony Field"
time=13:28:12 level=DEBUG msg="CTA found: swapping weapon and casting Battle Command / Battle Orders"
time=13:28:12 level=DEBUG msg=Buffing...
time=13:28:15 level=DEBUG msg="Reached area" area="Stony Field"
time=13:28:15 level=DEBUG msg="No items detected, waiting a bit and will try again" waitMs=1000
time=13:28:16 level=INFO msg="Run TerrorZone Run: Stony Field, Jail Level 1, Jail Level 2, Jail Level 3 finished, length: 13.20s" ```

I have similar problem.
TZ is in Arreat Plateu. I noticed that my nova sorc stucks a lot of times on the map, spelling a lot of nova when there is a lot of fire in area and doesnt teleport to new place. So I wanted to disable Arreat Plateu in config. After that bot creates new game, sorc goes to Arreat WP, casts all  buffs and quits game.
time=11:53:46 level=INFO msg="Waiting for character selection screen..."
time=11:54:02 level=INFO msg="Character selection screen found"
time=11:54:02 level=INFO msg="Selecting character..."
time=11:54:02 level=INFO msg="Character found"
time=11:54:06 level=DEBUG msg="Fetching map data..."
time=11:54:09 level=DEBUG msg="Fetch completed" ms=3708
time=11:54:09 level=INFO msg="Starting Game #0. Run list: TerrorZone Run: Arreat Plateau, Hell2, pit"
time=11:54:09 level=INFO msg="Running: TerrorZone Run: Arreat Plateau, Hell2"
time=11:54:10 level=DEBUG msg="Loading screen detected, waiting until loading screen is gone"
time=11:54:12 level=DEBUG msg="Load completed, continuing execution"
time=11:54:12 level=DEBUG msg="Checking for character corpse..."
time=11:54:12 level=DEBUG msg="Character corpse not found :D"
time=11:54:12 level=DEBUG msg="Checking for items to stash..."
time=11:54:12 level=DEBUG msg="Checking for items to identify..."
time=11:54:12 level=DEBUG msg="No items to identify..."
time=11:54:12 level=DEBUG msg="Belt Stats Health: 4/4 healing, 3/4 mana, 8/8 rejuv."
time=11:54:12 level=DEBUG msg="Checking for items to stash..."
time=11:54:12 level=DEBUG msg="Checking for items to stash..."
time=11:54:12 level=DEBUG msg="TZ area skipped" area="Arreat Plateau"
time=11:54:16 level=DEBUG msg="CTA found: swapping weapon and casting Battle Command / Battle Orders"
time=11:54:16 level=DEBUG msg=Buffing...
time=11:54:20 level=DEBUG msg="Reached area" area="Arreat Plateau"
time=11:54:20 level=DEBUG msg="Clearing TZ area" area=Hell2
time=11:54:20 level=DEBUG msg="Destination area not found" area=Hell2
time=11:54:20 level=WARN msg="Game finished with errors, reason: unrecoverable error occurred, game can not continue: attempt limit reached on step: InteractEntranceStep: area Hell2 [126]  not found. Game total time: 10.78s" supervisor=MoooMooo mapSeed=1502982702
time=11:54:23 level=INFO msg=Stopping... configuration=MoooMooo
time=11:54:23 level=INFO msg="Finished stopping" configuration=MoooMooo
time=11:54:30 level=ERROR msg="Error creating new game: error creating game! Timeout"


Add support for mender-partuuid

I've been trying to get Secure Boot implemented on a system that uses Mender for updates over USB. Following the very thorough examples I was able to add meta-mender-kernel and the required layers and configuration changes.
Unfortunately, we use mender-partuuid and we receive the following error:
ERROR: mender-kernel does not currently support mender-partuuid
Is mender-partuuid support in a planned release?

This is a restriction from meta-mender-core
        if [ ${partuuid} = "true" ]; then
            bbfatal "MENDER_EXTRA_PARTS and DISTRO_FEATURE 'mender-partuuid' are both defined and incompatible"
        fi

https://github.com/mendersoftware/meta-mender/blob/e0b2faf307a621cb1847c2991aa26016b3854f23/meta-mender-core/classes/mender-setup-image.inc#L64

I haven't put much thought into it, as this fit my use case OK. I'm not opposed to supporting it.... though it'd require PRs in meta-mender.

@Alvenix thanks for the PRs. First glance looks OK to me. I should have time this week to do a build and confirm.

@Alvenix thanks for the PRs. First glance looks OK to me. I should have time this week to do a build and confirm.

Thanks! I would like to note that for me I had to do fresh builds to test correctly. (It seems to me, caching doesn't play nice with meta-bitbake-variable-substitution)


Script Creating Empty Docker Images

Hello!
I am running the script in Unraid as shown for docker execution, but it seems that every time the script is run, it creates a new docker container with a strange name. (Ex: busy_zhukovsky, charming_antonelli, frosty_bouman, etc).
Each of these containers are dead containers and don't do anything, but they fill up my docker image on Unraid, and since I run the script daily on a schedule, it is creating a new container for each run. So there are 7 empty containers per week.
Any thoughts on why this is happening?

seems like your not naming the container in the launch command those weird names are default container names. Im inexperienced with Unraid but how are you running the script? are you using the -r option? If youre not using the -r option you should be able to just leave the script running and it will kick off once a day at 3am

Hello!
I am running the script in Unraid as shown for docker execution, but it seems that every time the script is run, it creates a new docker container with a strange name. (Ex: busy_zhukovsky, charming_antonelli, frosty_bouman, etc).
Each of these containers are dead containers and don't do anything, but they fill up my docker image on Unraid, and since I run the script daily on a schedule, it is creating a new container for each run. So there are 7 empty containers per week.
Any thoughts on why this is happening?

I'm running on unraid and do not have this issue, can you share your config screen?


Add schedule params & priority pthread functions

Fixes #609. Provides bindings to pthread_getschedparam, pthread_setschedparam and pthread_setschedprio functions.
P.S. My first PR in such a project.
P.P.S. libc-test crashed at building stage with a lot of errors, so I hope travis will do a check.
P.P.P.S. These functions are tested to work on Linux (Ubuntu 14.04) with my own crate. I will release it next week, it provides ability to control thread schedule policies and priorities.

I see the problem but will provide a fix only next week...

Looks good to me! I'll r+ once CI is green

As from what I have found in the Google setschedprio is not available at osx, bsd and android.
Also, I have filled an issue with my libc-test crate failures.

@alexcrichton even though this was compiled successfully I am still not sure did I find the correct place for it or no. It seems each system decides own rules how to follow POSIX :/ But from what I have seen I think pthread_setschedprio is linux-only function.

Ok this seems good for now regardless, thanks for the PR!
@bors: r+

:pushpin: Commit fbec8eb has been approved by alexcrichton

:hourglass: Testing commit fbec8eb with merge c25d657...

@alexcrichton I see that bors takes the whole first message here as a commit message. I think it should be changed, am I right?

:sunny: Test successful - status-appveyor, status-travis
Approved by: alexcrichton
Pushing c25d65702933faa92010c81b1630683853addf82 to master...

Ah no worries!


Fix Dockerfile so it runs

#27  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @vegarsti and the rest of your teammates on  Graphite