Rival Schools (US) and Shiritsu Justice Gakuen 2 (JP) show only black screen

tonyhax version: 1.4.3 freepsxboot via memory card slot 2
Installation method: PS2 via Memory Card Annihilator 2.0
Entry point game: NA
Console model: SCPH-5501 + SCPH-101 (PSone)
Integrity check: yes
BIOS version: 3.0 (SCPH-5501) & 4.5 (SCPH-101)
Target game:
Shiritsu Justice Gakuen - Nekketsu Seishun Nikki 2 (Japan) (SLPS-02120) ORIGINAL RETAIL
Rival Schools (SLUS-00681) BURNED VERBATIM AZO CDR, 16x with IMGBURN
Bug explanation:
On both my consoles, tonyhax recognizes and loads both games but I only get a black screen. Not capcom logo, no music.
Note: I burned gameshark 4.0 to act as a boot disc and that succeeded on the SCPH-101, but gameshark doesn't load properly on my 5501. Would really prefer to play this on the 5501, without a gameshark.

UPDATE: using Unirom 8.0.1 with freepsxboot 2.0 the games load just fine. It's odd that tonyhax isn't working then.

16x speed isn't the best to burn PlayStation discs. I highly recommend that you burn the disc at the slowest speed if possible, because the PSX does not like reading discs that were written at faster speeds.

16x speed isn't the best to burn PlayStation discs. I highly recommend that you burn the disc at the slowest speed if possible, because the PSX does not like reading discs that were written at faster speeds.

Especially older PSX console models, they really, really hate 16x. My results where unreliable when buring at 16x. 10x is the highest I will go with PSX CD-Rs, and luckily that is the minimum supported burn speed for my burner.
It is unfortunate that newer CD burners can't burn lower then 16x though.
The brand of CD-R is really good for PSX though, that is exactly what I use and recommend (Verbatim DataLifePlus CD-Rs).
It would be worth burning at a slower speed and trying again I would think. I'm going to try to reproduce this on real hardware.

16x speed isn't the best to burn PlayStation discs. I highly recommend that you burn the disc at the slowest speed if possible, because the PSX does not like reading discs that were written at faster speeds.

It sounds like the advice that you'd get from a YouTube tutorial, and that's because it is. I've seen many tutorials that recommend burning the CD at either max speed or 16x speed in "ImgBurn", as if that is going to ever work out well and I've had zero success with it.

This is reproducible. On my stock SCPH-1001 using Tonyhax v1.4.4 Boot CD via CD Player Swap Trick Rival Schools (SLUS-00681) boots into a black screen. I wonder what the GameShark is doing differently.

I'll see if its reproducable on one of my consoles, albeit I highly doubt it since NTSC games with AP doesn't activate on PAL consoles.

I'll see if its reproducable on one of my consoles, albeit I highly doubt it since NTSC games with AP doesn't activate on PAL consoles.

That's not always the case. Also I don't think this is AP related because there is no anti-piracy screen that is triggered.

I'll see if its reproducable on one of my consoles, albeit I highly doubt it since NTSC games with AP doesn't activate on PAL consoles.

That's not always the case. Also I don't think this is AP related because there is no anti-piracy screen that is triggered.

Shiritsu Justice Gakuen 2 does have anti-modchip protection according to Redump.

I'll see if its reproducable on one of my consoles, albeit I highly doubt it since NTSC games with AP doesn't activate on PAL consoles.

That's not always the case. Also I don't think this is AP related because there is no anti-piracy screen that is triggered.

Shiritsu Justice Gakuen 2 does have anti-modchip protection according to Redump.

As the game does not boot at all, it might not be related to that. It's possible that the anti-piracy code never even got a chance to run.

Interesting, Rival Schools works in Tonyhax International v1.1.3 but not in Tonyhax v1.4.4 on the same console.

Same happened with Shiritsu Justice Gakuen 2. Worked fine on Tonyhax Int. v1.1.3 but not on Tonyhax v1.4.4.

Same happened with Shiritsu Justice Gakuen 2 as well as Rival Schools. Worked fine on Tonyhax Int. v1.1.3 but not on Tonyhax v1.4.4.

I compiled the latest source from github (as of 12/1/2022) of Tonyhax (v1.4.5b) and this is still an issue.
Hmm, the 'loader' diff from Tonyhax International comparing the original Tonyhax is here:  https://github.com/alex-free/tonyhax/blob/master/docs/tonyhax-vs-tonyhax-international-diffs/tonyhax-international-v1.1.3.diff

I cannot replicate this on an emulator, and I cannot test it in real hardware at the moment. Can some of you attempt to run it with patches disabled, please?

I cannot replicate this on an emulator, and I cannot test it in real hardware at the moment. Can some of you attempt to run it with patches disabled, please?

BOOM, it works! Attached is the latest Tonyhax v1.4.5b compiled directly from Github, with one small change:
//patcher_apply(bootfile); (in the file secondary.c)
This makes a lot of sense. Tonyhax International uses a completely different anti-piracy bypass system, as well as the newer more compatible FreePSXBoot patches (from the FreePSXBoot builder itself: https://github.com/brad-lin/FreePSXBoot/commit/62287d1754efa5c1e131f9f36dbda6a89f39d4f3).
Now neither me or @M4x1mumReZ where using the FPSXBoot method, and I actually did try removing only the FreePSXBoot patch from Tonyhax and it didn't work. So this confirms the anti-piracy patch is interfering with these 2 games directly.

I really rather figure out why is the check causing the problem rather than to blacklist or whitelist it - the antipiracy does very plenty checks to ensure it does not cause an illegal or unaligned read, so I'm really curious on what the issue could be here.
I guess I'll revisit that idea of getting an old non-slim PS1 to load executables straight into RAM using the serial port, which should make testing this much much easier.

@M4x1mumReZ can you verify this Boot CD works as well?

I'll do that right now and I'll let you know.

Okay, I can confirm that both work now. 👍🏻

I've devised a way to load tonyhax with no memory card, only an Arduino. Furthermode, it's a full-fledged terminal so I can peek and poke arbitrary memory registers, which could be pretty useful for testing that theory about changing PS2-only video registers in PS1 mode:

I'll start now investigating these black screens of death.

 I've devised a way to load tonyhax with no memory card, only an Arduino. Furthermode, it's a full-fledged terminal so I can peek and poke arbitrary memory registers, which could be pretty useful for testing that theory about changing PS2-only video registers in PS1 mode:  I'll start now investigating these black screens of death.

Now that is interesting. 😄

Still working on this, but this new debugging feature is gonna make my life much, much more easy.

Thanks to the contraception, I've managed to easily figure out the issue. From the interrupt log:
Configuration : EvCB	0x10		TCB	0x04
Int PC:801FC098 SP:801FFE70
Change effective memory : 2 MBytes
Int PC:BFC0D968 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:00006B88 SP:801FFF78
Int PC:8005AD14 SP:801FFFA8
Int PC:8005AD14 SP:801FFFA8
Int PC:8005AD14 SP:801FFF20
Int PC:8005AD14 SP:801FFFD8 <<< Crashes here if we read SP+0x28
Int PC:8005AD14 SP:801FFFB8
Int PC:8005AD14 SP:801FF7C8
Int PC:8005AD14 SP:801FF7C8

SP+801FFFD8 is 80200000, which is just a mirror of the RAM and should not cause any harm. However, the game uses the SetMemSize function, which does cause the machine to lock up if accessing a mirror of the RAM.
Fixed in b70cbe6bc401311cc9f8f0722c2bba2021401509. Attached is a beta 1.4.5 which should work.
tonyhax-v1.4.5b.zip
Closing, but feel free to reopen if needed.

Thanks to the contraception, I've managed to easily figure out the issue. From the interrupt log:
Configuration : EvCB	0x10		TCB	0x04
Int PC:801FC098 SP:801FFE70
Change effective memory : 2 MBytes
Int PC:BFC0D968 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:00006B88 SP:801FFF78
Int PC:8005AD14 SP:801FFFA8
Int PC:8005AD14 SP:801FFFA8
Int PC:8005AD14 SP:801FFF20
Int PC:8005AD14 SP:801FFFD8 <<< Crashes here if we read SP+0x28
Int PC:8005AD14 SP:801FFFB8
Int PC:8005AD14 SP:801FF7C8
Int PC:8005AD14 SP:801FF7C8

SP+801FFFD8 is 80200000, which is just a mirror of the RAM and should not cause any harm. However, the game uses the SetMemSize function, which does cause the machine to lock up if accessing a mirror of the RAM.
Fixed in b70cbe6. Attached is a beta 1.4.5 which should work. tonyhax-v1.4.5b.zip
Closing, but feel free to reopen if needed.

Any way you could publish this tool of yours?

Thanks to the contraception, I've managed to easily figure out the issue. From the interrupt log:
Configuration : EvCB	0x10		TCB	0x04
Int PC:801FC098 SP:801FFE70
Change effective memory : 2 MBytes
Int PC:BFC0D968 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:8005AD14 SP:801FFF90
Int PC:00006B88 SP:801FFF78
Int PC:8005AD14 SP:801FFFA8
Int PC:8005AD14 SP:801FFFA8
Int PC:8005AD14 SP:801FFF20
Int PC:8005AD14 SP:801FFFD8 <<< Crashes here if we read SP+0x28
Int PC:8005AD14 SP:801FFFB8
Int PC:8005AD14 SP:801FF7C8
Int PC:8005AD14 SP:801FF7C8

SP+801FFFD8 is 80200000, which is just a mirror of the RAM and should not cause any harm. However, the game uses the SetMemSize function, which does cause the machine to lock up if accessing a mirror of the RAM.
Fixed in b70cbe6. Attached is a beta 1.4.5 which should work. tonyhax-v1.4.5b.zip
Closing, but feel free to reopen if needed.

Any way you could publish this tool of yours?

Believe it was this tool: https://github.com/socram8888/joymon

That is correct. Tonyhax has also a new variable in variables.mk which you can use to enable it while running a commercial game. It replaces the std_out_putc so all printfs, even from the kernel or the game itself, are visible.
It is disabled by default since there's little reason for the average Joe to enable it, and could introduce some compatibility issues.

That is correct. Tonyhax has also a new variable in variables.mk which you can use to enable it while running a commercial game. It replaces the std_out_putc so all printfs, even from the kernel or the game itself, are visible.
It is disabled by default since there's little reason for the average Joe to enable it, and could introduce some compatibility issues.

This can be useful in many cases to debug and look for AP routines.


Packit-as-a-Service is on GitHub Marketplace. Fix docu

Signed-off-by: Petr "Stone" Hracek<EMAIL_ADDRESS>Since Packit-as-a-Service is available on GitHub Marketplace (https://github.com/marketplace/packit-as-a-service) we can use reference only to it.

Make sense. We have the issue for it https://github.com/packit-service/packit.dev/issues/27


Support custom element attributes with checking

item.ts
import {bindable} from "aurelia-templating";

export class ItemCustomElement {
    @bindable value: string;
}

item.html
<template>
    ${value}
</template>

page.ts
export class PageViewModel {
    value: number;
}

page.html
<template>
    <require from="./item"></require>

    <item title.bind="value" value.bind="value"></item>
</template>

This should throw two errors, the first for missing bindable title attribute in ItemCustomElement, the second for mismatching type of value attribute.
Update: If --strictNullCheck is enabled, it could also check if all required bindable attributes are passed, meaning only those explicitly typed with X | undefined could be left unassigned.

@MeirionHughes, currently this issue is assigned to 0.11 milestone, but i think that this issue has actually the most impact for this project (mostly for newcomers, but it could have saved some of my time as well last week because of copy-past from camelCaseProperty that should have been converted to kebab-case when used as attribute)

Unfortunately, the rework stuff is slow going (0.10) as its taken a while to decide how to structure it --I'm favoring chain-of-responsiblity style for the rules that operate on the ast - thus abstracting away the parser entirely.
So... I will take a look at this to see if something can be done asap.

Im I correct that currently the missing prerequisite for this task is ability to check/resolve references to other files via <require from="..."?
So probably a components that would

detect (or in initial phase just provide) global resources
scan references to the source files
register custom element tag names

would be smth in the beginning of the responsiblity chain? Is that smth You have in mind?

Yeah something like that; basically you pass the file result along and they add extra processed data to the result/ast. not entirely there yet. the idea is to make it generic enough so I can throw any file type at the chain.
However, for now, I could probably to grab the source when <require from='./item'' comes up. see if there the first export has the custom element decorator, define a custom element resource, and finally, attach the resource to the ast. then when checking the bindings we check:

I created a checklist (containing two ideas from @zakjan and added couple more):

[ ] invalid path references: <require from="./does/not/exist"></require>
[ ] that attribute are not in camelCase, but in kebab-case format: <item notCorrect.bind="existing"></item> vs <item is-correct.bind="existing"></item> ( subset of next check, but might be easier to implement, so I added it separately )
[ ] missing bindables <item missing.bind="existing"></item>
[ ] mismatching type of value <item some-number.bind="someString"></item>

Another idea: If --strictNullCheck is enabled, it could also check if all required bindable attributes are passed, meaning only those explicitly typed with X | undefined could be left unassigned.

okay-doky, I've added something basic. its limited to './foo' having a single class with FooSomeThingCustomElement and the resulting element being foo-some-thing.

More ideas:

value.bind="..." binding syntax should be required, only string attributes should allow passing value with value="..."
If strictNullChecks is enabled, only nullable attributes (@bindable value: ... | undefined) can be omitted, otherwise they are required.


Woff2 support

Are there any further plans for updates on this plugin?
I'm happily using it, but now starting to miss woff2 support.
Seems fairly trivial to add. I could create a pull request if interested.

@tylerbeck Are you still working on this? And if not, would you have any suggestion for an alternative?


loader/layers: Fix MSVS warnings

A couple warnings popped up with my integration's build. These are fairly low impact. Hopefully the new enable flags will prevent further regressions (and might catch a few in the CI builder that I didn't fix).

Looks like there are a couple more warnings. #2195 should fix two of them, but there's one other warning in the layers.


Debug console not appending output properly

process.stdout.write("hello");
process.stderr.write("world");

Debug console prints these in two lines which is unexpected

@weinand this works as expected for me, since these writes are sent to different streams - first is stdout second is stderr. We seperate the writes based on severity - e.g first is info, second is error (which is in colored red). The following snippets prints everything in one line as expected
process.stdout.write("hello");
process.stdout.write("world");

@isidorn your example prints everything in one line because you only get one OutputEvent, correct? The node runtime buffers writes. So your example doesn't prove that no additional newline is appended.
The purpose of my example from above was to prevent the buffering by using different streams and prove that an additional newline is appended.
In the integrated terminal or the external terminal both strings appear concatenated in one line.
In the debug console I see a newline after the first string, so the strings appear in two lines.
A terminal or debug console is a single linear stream of characters (as can be seen in the integrated or external terminals). Whether the characters come from stdout or stderr should not have an influence on how the characters are distributed on different lines, only the used color is affected.
Why do you have to start a new line when OutputEvents change their channel?

@weinand this example also writes everything in one line. I am getting two output events
process.stdout.write("hello");
setTimeout(() => process.stdout.write("world"), 5000);

We do not have to start a new line when OutputEvents change their channel. But we do it because we color the output channels differently. And currently we do not have support to color parts of the line red. Also imho we should somehow indicate to the user that the output differs.
Let me know if you would prefer that everything goes to one line and I can make that change.
Also FYI @bpasero since he was invovled in that code also

@isidorn we should show different colours for different channels and we should render in the same way as the integrated or external terminals.
If we do not yet have support to color parts of a line, we can wait until debug-console@next supports this.


Crypt.php Microaudit

Time-boxing an audit of Crypt.php limited to 20 minutes:

In _generator: There is absolutely no use to combining mt_rand() and microtime() with mcrypt_create_iv(). In fact, every extra step you do is only increasing the likelyhood of mistakes causing the result to be less random. Just use mcrypt_create_iv() and nothing more. Keep it simple.

What I hypothesized in (1) did indeed happen. First, a specific problem: The call to hash() for $seedThree returns the hash as a hex, not in binary. So $seedThree will take on only hex character values (0-9a-f), so some of the output is going to be hex characters, and not random bytes. A more general problem: The bytes generated by the other techniques have very low entropy. That means in the output there are likely to be low entropy bytes. Because of the shuffle, (which is also low entropy) an attacker may not know where those bytes are, but  that doesn't matter because they are still there. Again, just use mcrypt_create_iv().

There is no padding applied to the plaintext before encryption. That means you'll get extra null bytes in the plaintext after decryption (mcrypt_encrypt pads with nulls).

The IV is not included in the hash. An attacker could modify what the plaintext decrypts to by changing the IV (this breaks confidentiality as well, in some cases).

The decrypt() function doesn't properly validate the ciphertext format. Most of the calls to substr() just assume the string has the correct length, when it might not.

The way it derives $mackey from $key is not standard, it should be doing HKDF.

In generateRandomString(): The line $o = ord($random[$i]) % $len; introduces heavy bias into the output because of the % unless $len divides 256 (it doesn't).

In generateRandomInt(), it takes the strlen() of an integer $range, which makes no sense to me. That's not the right way to do it. Here is an example of how to do it correctly.

In generateRandomInt(), well... the whole process makes no sense and is probably very biased. See the above link for the correct algorithm for doing this.

Conclusion: This code is very broken and I recommend scrapping it. Replace it with:
For encryption: https://github.com/defuse/php-encryption/
For password generating: https://github.com/defuse/php-passgen/blob/master/PasswordGenerator.php
Both of the above libraries are very carefully written, and I suggest at least studying them to learn more about how hard it is to do this right. Please do not write your own crypto.

Hi Taylor,
thank you for responding to our review request, we much appreciate your feedback.
We did look at the libraries you wrote, and we were interested in integrating them, instead of rolling our own, but unfortunately those libraries are not available on composer, which make is impossible for us to use them, and also licensing of those components is not specified. If the components get those updates, we will definitely be interested in integrating them.
For the time being we will take into account your feedback and correct the noted issues.
I do thank you again, we really appreciate when somebody is willing to step up and give honest and concrete feedback.

@SvenAlHamad I will prioritize getting those libraries on composer so that they are more easy to use. Crypto.php is under the MIT license, and I will apply the MIT license to the password generating one as well.

Crypto.php is now in composer.

php-encryption: https://packagist.org/packages/defuse/php-encryption
php-passgen: https://packagist.org/packages/defuse/php-passgen

@defuse Thank you for getting the libraries to composer, I did manage to install them.
Looking at the library there are still some issues that prevent us from using the library:
php-encryption:

exception handler is globally overwritten
key needs to have 16 bytes - not sure why is that, why don't you internally hash the given key to the requested length and use that for encryption, that way the key can be of any length
missing namespaces

php-passgen:

PasswordGenerator.php has a closing php tag, that can cause a lot of issues when returning output to browser
doesn't support generating random int between min and max int

I see that some of them are already reported on your repo. I believe once those issues are resolved, a lot members will be looking at the library, including us, but for the time being that is preventing us from using them.
I've added myself to the watchlist, and will keep an eye on the progress.
I do thank you again on your effort and contribution.

@SvenAlHamad Regarding the key size, it only accepts 16 byte keys because that is what AES accepts. If the user is passing in an alphanumeric string of variable length they are almost certainly using the library wrong. It is possible to accept a string of any length, but then it would be called a passphrase and there would be an additional step of key stretching to create the 16 byte key from it (this is a planned feature for Crypto.php).

I removed the closing tag from PasswordGenerator.php and added an (untested!) random int function. Will probably get around to addressing the other issues in the next few weeks. Thanks for the feedback!

@SvenAlHamad What do you think about this API for supporting passphrases (keys that are not random 16 bytes): https://github.com/defuse/php-encryption/issues/5#issuecomment-34369429 The reason an API like that is necessary is because the KDF function needs a random salt.
I strongly recommend just sticking to random 16-byte keys, though.

I get that 16-byte keys are better, when it comes to the 'code-level', but it makes the public api less usable and developer friendly. The key should be making the library easy to use, without compromising the security aspect of the component itself.
The proposed KeyLocker class should probably solve that problem.
For now, I've resolved the issues that you mentioned in your audit report, until your libraries are ready.

Closing for now ... until the php-encryption and passgen tasks have been resolved.


Using symbols or special characters in labels

I would like to use:
    label: "Félix"

or
    label: "Félix"

in the Schema, but it renders like that in autoform.
Any clue?

Hey @reel, there's still no way of doing that

Sorry, I am no longer using meteor

Thanks for anwering so fast, I'm still looking for a solution :-/

I don't remember what I did or if I found a solution. Let me know if you want me to reopen this issue

Well, that would be just great, thanks


Card heights and embeds

Any time a note has an ![[embed]] it stretches the card the full height of the embed.
Other cards seem to be so small they only display a line or two. While yet others display a little bit more reasonably.

I updated the plugin and unfortunately this is still an issue. Any note with an ![[embedded]]] note will stretch the card the entire length of that note.

I promise it will be fixed sooner or later, I just cannot guarantee you when, because I maintain this plugin on my spare time.

No worries, looking forward to using your plugin :)


[feature-request] export cucumber-related types

What are you trying to achieve?
export DataTable and other cucumber-related types such that we can use it in step definitions
What do you get instead?
I need to manually define it on our own
interface OwnTableDefinition {
  /**
   * obtain an object that allows you to get a simple version of the table parsed by column or row
   */
  parse(): {
    /**
     * returns the table as a 2-D array
     */
    raw(): string[][],
    /**
     * returns the table as a 2-D array, without the first row
     */
    rows(): string[][],
    /**
     * returns an array of objects where each row is converted to an object (column header is the key)
     */
    hashes(): { [colName: string]: string }[],
  };
  rows: { [id: number]: { cells: { value: string }[] } };
}

Given('I have products in my cart', (table: OwnTableDefinition) => {
   table.parse()
})

Details

CodeceptJS version: 3.0.7
NodeJS Version: 12.20.1
Operating System: MacOS

Any volunteers to send pull request?

Feel free to send PR @zaxoavoki 😃


Lightspeed ignores bigrams prefixed with

Pressing s/ yields the following result

As you can see, only one occurence of // gets highlighted. However, if I set list, the result is this

With set list on, everything works fine. Turning it off again produces the buggy first behaviour again.
Notice how the first occurence of // is prefixed by a space, the others by a tab. I thought this might be the sole reason for this bug, but I wasn't able to reproduce that behaviour in a simple text file.
So I copied a small chunk of the C code into a text file, in which I am able  to reproduce that behaviour. I'm attaching said file to this issue so you can use it for debugging.
This doesn't only happen when targeting the //, the same issue persists targeting the lines starting with sConfig.
Pressing ss with set nolist yields

And again, with set list it behaves like expected.

My lightspeed config is completely stock, I've only installed it but changed no settings.
lightspeed_bug_repro.txt
Edit:
NVIM v0.6.1-dev+656-g84784a839
Build type: Release
LuaJIT 2.1.0-beta3
Compiled by js@tmt

Features: -acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM\sysinit.vim"
  fall-back for $VIM: "C:/Program Files (x86)/nvim/share/nvim"

Run :checkhealth for more info

~/temp ❯ [Environment]::OSVersion

Platform ServicePack Version      VersionString
-------- ----------- -------      -------------
 Win32NT             10.0.19042.0 Microsoft Windows NT 10.0.19042.0

Thanks for the thorough bug report! I cannot reproduce this. I tried this with nvim -u NONE + manually sourcing lightspeed.lua too, and it still works, so it's probably something in your config. That might still indicate some bug to be fixed on our side, but this is good news, because sooner or later it can be hunted down then. I have no idea where to start though.

Okay, I got it.
1.:
Use the following config
call plug#begin('~/.vim/plugged')
Plug 'ggandor/lightspeed.nvim'
call plug#end()
setlocal nowrap

2.:
nvim lightspeed_bug_repro.txt
3.:
press ss
This will yield the following result:

4.:
set list make lightspeed work as expected, ie:

Ouch, I can reproduce this now. Indeed, it has something to do with tabs, if you replace them with spaces, works fine.

yeah, seems it does. I tried reproducing this artificially though and couldn't do it so far. I'm now working with the file I uploaded here and trying to have the smallest amount of content that triggers the bug. It's not as simple as having mixed indents, I tried that already.
Do you have any idea what might be causing this?

Okay, so, it get's better/worse. Depends how you look at it. With this file, place the cursor at the beginning of the first line. Then press s/.
That causes my entire neovim instance to lock up until I press ctrl+c. Can you reproduce that aswell?
minimal_repro.txt

Can you reproduce that aswell?

No :) But now I cannot reproduce again on the original file either...

Do you have any idea what might be causing this?

A very wild guess, might be a total misfire: this could be related to the calculation of the leftmost editable screen column (we should ignore the fold column, line numbers, etc.), maybe the search loop skips those matches for some reason? Does the problem appear anywhere in the middle of the window, when the target is not in the first few columns?

Okay, I got this, just as I thought. The root of the problem is that we do a hack to calculate the "gutter" width, by doing a norm! 0, and check the column we're in, but if the first char is a tab, that will put the cursor to the last (screen) column the tab occupies, e.g. if tabstop=8, then to the 8th column in the actual editable area, not the 0th, and that messes up the whole subsequent calculation, so the first 8 char will be skipped on each line.

ah, the ninja edit, now my comment makes no sense :P

Okay, I got this, just as I thought. The root of the problem is that we do a hack to calculate the "gutter" width, by doing a norm! 0, and check the column we're in, but if the first char is a tab, that will put the cursor to the last (screen) column the tab occupies, e.g. if tabstop=8, then to the 8th column in the actual editable area, not the 0th, and that messes up the whole subsequent calculation, so the first 8 char will be skipped on each line.

Alright, sounds like you know what to look for. Let me know if I need to to anything.
I was just in the middle of testing to confirm that the tabstop setting does influence this somehow. But you are right, it does.

Alright, sounds like you know what to look for.

Yeah, at the very latest, it will be fixed when the improved getwininfo() will work properly, and I can use that to get rid of the norm! 0 hack (in theory, on 0.6. it should be able to return the necessary info in the textoff field, but it doesn't work for me at the moment).

Seems to work fine so far, I've used this at work today and haven't encountered any issues yet.


What to do with shared volumes exposed as raw block devices inside a pod?

I could expose a shared disk as a raw block device inside a pod, but then what?!? lsblk doesn't show it as device, I could partition it but the new primary partition is not visible as device either, mounting is out of question. What in the (edited) world am I supposed to do with this raw block, keep it for decoration? Probably I am missing some knowledge, but then again, isn't this the point of this document to provide me with this knowledge?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 4ba9a74f-e0b7-5e2c-b066-5f32dc47f950
Version Independent ID: 919e9aef-832f-574f-89b3-0456c5cc3f68
Content: Use Container Storage Interface (CSI) drivers for Azure Disks on Azure Kubernetes Service (AKS) - Azure Kubernetes Service
Content Source: articles/aks/azure-disk-csi.md
Service: container-service
GitHub Login: @palma21
Microsoft Alias: zarhoads

@patuf Edited for language.

@patuf I've assigned this to the document author for further review to see if there is something we can add to the documentation to add further clarification.

Hi @patuf, thanks for your feedback. Please review https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/app-platform/aks/storage#disk-based-solutions to see disk-based solutions and let me know if the resources help.


Consistent payload validation

KNTK - 386
https://clkentik.atlassian.net/browse/KNTK-386
All payload validation located in /api_resources moved to requests_payload.
To make this validation consistent also refactored create/update classes in /api_resources.

I'll wait for @danielfurman  to approve.


FDG-4933 Line chart with multiple scales

https://federal-spending-transparency.atlassian.net/browse/FDG-4933
Includes implementation (FDG-4698).
https://federal-spending-transparency.atlassian.net/browse/FDG-4698
Some linting fixes, page layout/wrapping adjustments
Test coverage: 89.02

@wyattch , I see the point about the header spacing, but I don't  notice a blue line or a black hatching.  I this how it looks on your screen?


Docker build is failing on npm install --production command

Deploy script is failing where it tries to build a docker image at this line.
docker build -t "cloudformation-lambda-$function" .
The error is:
The command '/bin/sh -c npm install --production' returned a non-zero code: 132.
In Dockerfile, this is the particular line which is failing:
RUN npm install --production
I tried googling it but none of the proposed solutions is working. Please help.

Can you try changing that npm install line to this to see if you get more details about the error?
RUN npm install --production --loglevel verbose

Thanks for the quick response. Since this is happening on docker container, I tried running the container in the interactive mode and run the command npm install --production --loglevel verbose. Got the following error:
Illegal instruction (core dumped)
Before doing above, I modified the docker file as you suggested and run the build command. Here is the entry from /var/log/syslog file:
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024952.718497] docker0: port 2(vethd1855f6) entered blocking state
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024952.718499] docker0: port 2(vethd1855f6) entered disabled state
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024952.718547] device vethd1855f6 entered promiscuous mode
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024952.718642] IPv6: ADDRCONF(NETDEV_UP): vethd1855f6: link is not ready
Dec 31 16:19:21 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-resolved[15285]: message repeated 5 times: [ Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.]
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[8935]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[8935]: Could not generate persistent MAC address for vethb29fb6f: No such file or directory
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[8940]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[8940]: Could not generate persistent MAC address for vethd1855f6: No such file or directory
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.0261] manager: (vethb29fb6f): new Veth device (/org/freedesktop/NetworkManager/Devices/357)
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.0273] manager: (vethd1855f6): new Veth device (/org/freedesktop/NetworkManager/Devices/358)
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.0702] devices added (path: /sys/devices/virtual/net/vethb29fb6f, iface: vethb29fb6f)
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.0702] device added (path: /sys/devices/virtual/net/vethb29fb6f, iface: vethb29fb6f): no ifupdown configuration found.
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.0788] devices added (path: /sys/devices/virtual/net/vethd1855f6, iface: vethd1855f6)
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.0788] device added (path: /sys/devices/virtual/net/vethd1855f6, iface: vethd1855f6): no ifupdown configuration found.
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC containerd[942]: time="2019-12-31T16:21:00.533461197-06:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/e08908194abb9fb31fdc7907f7f340866a294e8722bc1833f28f2ce7b947ad9d/shim.sock" debug=false pid=9005
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024953.578821] eth0: renamed from vethb29fb6f
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024953.594550] IPv6: ADDRCONF(NETDEV_CHANGE): vethd1855f6: link becomes ready
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024953.594616] docker0: port 2(vethd1855f6) entered blocking state
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024953.594617] docker0: port 2(vethd1855f6) entered forwarding state
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.8909] devices removed (path: /sys/devices/virtual/net/vethb29fb6f, iface: vethb29fb6f)
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830860.8920] device (vethd1855f6): carrier: link connected
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[3434]: Removing a network device that was not added
Dec 31 16:21:00 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[1600]: Removing a network device that was not added
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024954.092136] traps: node[9045] trap invalid opcode ip:7fd0c6fdcb30 sp:7ffef6e0d358 error:0 in libstdc++.so.6.0.20[7fd0c6f7d000+11e000]
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC containerd[942]: time="2019-12-31T16:21:01.713681861-06:00" level=info msg="shim reaped" id=e08908194abb9fb31fdc7907f7f340866a294e8722bc1833f28f2ce7b947ad9d
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC dockerd[32725]: time="2019-12-31T16:21:01.723683660-06:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024954.495272] docker0: port 2(vethd1855f6) entered disabled state
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024954.497380] vethb29fb6f: renamed from eth0
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[9192]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830861.8357] manager: (vethb29fb6f): new Veth device (/org/freedesktop/NetworkManager/Devices/359)
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830861.8651] devices added (path: /sys/devices/virtual/net/vethb29fb6f, iface: vethb29fb6f)
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830861.8651] device added (path: /sys/devices/virtual/net/vethb29fb6f, iface: vethb29fb6f): no ifupdown configuration found.
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024954.628262] docker0: port 2(vethd1855f6) entered disabled state
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024954.635323] device vethd1855f6 left promiscuous mode
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024954.635330] docker0: port 2(vethd1855f6) entered disabled state
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830861.9469] devices removed (path: /sys/devices/virtual/net/vethb29fb6f, iface: vethb29fb6f)
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830861.9530] device (vethd1855f6): released from master device docker0
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[1600]: Removing a network device that was not added
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[1600]: Removing a network device that was not added
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[3434]: Removing a network device that was not added
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[3434]: Removing a network device that was not added
Dec 31 16:21:01 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830861.9739] devices removed (path: /sys/devices/virtual/net/vethd1855f6, iface: vethd1855f6)
Dec 31 16:21:02 faisal-HP-Compaq-6005-Pro-SFF-PC dockerd[32725]: time="2019-12-31T16:21:02.122304464-06:00" level=warning msg="e08908194abb9fb31fdc7907f7f340866a294e8722bc1833f28f2ce7b947ad9d cleanup: failed to unmount IPC: umount /var/lib/docker/containers/e08908194abb9fb31fdc7907f7f340866a294e8722bc1833f28f2ce7b947ad9d/mounts/shm, flags: 0x2: no such file or directory"
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[9735]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[9735]: Could not generate persistent MAC address for vethc4f1918: No such file or directory
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.1851] manager: (vethc4f1918): new Veth device (/org/freedesktop/NetworkManager/Devices/360)
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[9736]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[9736]: Could not generate persistent MAC address for veth4e78759: No such file or directory
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.1886] manager: (veth4e78759): new Veth device (/org/freedesktop/NetworkManager/Devices/361)
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024987.896864] docker0: port 2(veth4e78759) entered blocking state
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024987.896867] docker0: port 2(veth4e78759) entered disabled state
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024987.897199] device veth4e78759 entered promiscuous mode
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024987.899853] IPv6: ADDRCONF(NETDEV_UP): veth4e78759: link is not ready
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.2205] devices added (path: /sys/devices/virtual/net/vethc4f1918, iface: vethc4f1918)
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.2205] device added (path: /sys/devices/virtual/net/vethc4f1918, iface: vethc4f1918): no ifupdown configuration found.
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.2322] devices added (path: /sys/devices/virtual/net/veth4e78759, iface: veth4e78759)
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.2322] device added (path: /sys/devices/virtual/net/veth4e78759, iface: veth4e78759): no ifupdown configuration found.
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC containerd[942]: time="2019-12-31T16:21:35.711892623-06:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/02785ec5f6be9261ba3d4576630262fb47f16e2696d24f3e625fea910af4fd2d/shim.sock" debug=false pid=9770
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024988.671905] eth0: renamed from vethc4f1918
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[3434]: Removing a network device that was not added
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024988.691135] IPv6: ADDRCONF(NETDEV_CHANGE): veth4e78759: link becomes ready
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024988.691237] docker0: port 2(veth4e78759) entered blocking state
Dec 31 16:21:35 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024988.691238] docker0: port 2(veth4e78759) entered forwarding state
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[1600]: Removing a network device that was not added
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.9826] devices removed (path: /sys/devices/virtual/net/vethc4f1918, iface: vethc4f1918)
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830895.9877] device (veth4e78759): carrier: link connected
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024989.250920] traps: node[9800] trap invalid opcode ip:7f9df7a29b30 sp:7ffd877f20a8 error:0 in libstdc++.so.6.0.20[7f9df79ca000+11e000]
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC containerd[942]: time="2019-12-31T16:21:36.863163653-06:00" level=info msg="shim reaped" id=02785ec5f6be9261ba3d4576630262fb47f16e2696d24f3e625fea910af4fd2d
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC dockerd[32725]: time="2019-12-31T16:21:36.873382068-06:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024989.632940] docker0: port 2(veth4e78759) entered disabled state
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024989.634198] vethc4f1918: renamed from eth0
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC systemd-udevd[9882]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830896.9654] manager: (vethc4f1918): new Veth device (/org/freedesktop/NetworkManager/Devices/362)
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830896.9941] devices added (path: /sys/devices/virtual/net/vethc4f1918, iface: vethc4f1918)
Dec 31 16:21:36 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830896.9941] device added (path: /sys/devices/virtual/net/vethc4f1918, iface: vethc4f1918): no ifupdown configuration found.
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024989.766059] docker0: port 2(veth4e78759) entered disabled state
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024989.777539] device veth4e78759 left promiscuous mode
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC kernel: [1024989.777548] docker0: port 2(veth4e78759) entered disabled state
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830897.0869] devices removed (path: /sys/devices/virtual/net/vethc4f1918, iface: vethc4f1918)
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830897.0908] device (veth4e78759): released from master device docker0
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[3434]: Removing a network device that was not added
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[3434]: Removing a network device that was not added
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[1600]: Removing a network device that was not added
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC gnome-shell[1600]: Removing a network device that was not added
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC NetworkManager[664]:   [1577830897.1220] devices removed (path: /sys/devices/virtual/net/veth4e78759, iface: veth4e78759)
Dec 31 16:21:37 faisal-HP-Compaq-6005-Pro-SFF-PC dockerd[32725]: time="2019-12-31T16:21:37.281064125-06:00" level=warning msg="02785ec5f6be9261ba3d4576630262fb47f16e2696d24f3e625fea910af4fd2d cleanup: failed to unmount IPC: umount /var/lib/docker/containers/02785ec5f6be9261ba3d4576630262fb47f16e2696d24f3e625fea910af4fd2d/mounts/shm, flags: 0x2: no such file or directory"

I don't have good advice for debugging further. Since this error appears to be related to npm install, I'd suggest asking for help in the npm forums: https://npm.community/c/support

OK. Thanks.

I'm able to resolve it. But I had to create 3 entirely new Dockerfiles. I am using amazonlinux instead of lambci. Please let me know if you want me to check those Dockerfiles into your repository.

Would you mind filing a PR? I may not merge it but it might be helpful if anyone else runs into this. Thanks!

It has been a while since I worked on Git and forgot the commands. Here is one of the Dockerfiles (site-builder). Should be good enough for people to create other if needed.
FROM amazonlinux
RUN yum install -y tar
RUN yum install -y gzip
RUN yum install -y zip
RUN curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.6/install.sh | bash
RUN /bin/bash -c "source /root/.nvm/nvm.sh; nvm install 6.11.5"
CMD /bin/bash -c "source /root/.nvm/nvm.sh; nvm use 6.11.5"
RUN /bin/bash -c "source /root/.nvm/nvm.sh; npm install --production"
RUN mkdir /build
RUN mkdir /data
WORKDIR /build
install dependencies (highly cacheable)
COPY package.json /build/package.json
RUN curl -L https://www.npmjs.com/install.sh | sh
RUN rm -rf node_modules
RUN npm cache clean
RUN export USER=root && npm install --production
RUN npm install --production --loglevel verbose
RUN /bin/bash -c "source /root/.nvm/nvm.sh; npm install --production"
add source code
COPY index.js /build/index.js
add HTML templates
COPY homepage /build/homepage
COPY album /build/album
zip entire context and stream output
RUN zip -r /build/dist.zip . > /dev/null
CMD ["cat", "/build/dist.zip"]

Thanks for posting! Let’s keep this issue open to make it easier for others to find.


error

it seems request.py is missing as
C:\Users\shikh\OneDrive\Desktop\telegramscraper-main>py manager.py
Traceback (most recent call last):
File "C:\telegramscraper-main\manager.py", line 1, in
import requests
ModuleNotFoundError: No module named 'requests'

pip install requests


rbf for mac

Hey Ethan,
any news on getting RBF to work on mac? I tried installing that version but had no luck.
Thanks

I don't think so because it requires the underlying library to support Mac. I'm unable to rewrite the library to make it work purely on Python either, so I don't think this possible for now.


New version: Bytez v0.0.6

Registering package: Bytez
Repository: https://github.com/Bytez-com/docs
Created by: @inf3rnus
Version: v0.0.6
Commit: 9e2e6c01abd8703bee2be2df9626caa1d517df2a
Reviewed by: @inf3rnus
Reference: https://github.com/Bytez-com/docs/commit/9e2e6c01abd8703bee2be2df9626caa1d517df2a#commitcomment-148813800
Description: Access 14k+ open source AI models across 30+ tasks with the Bytez inference API ✨
Release notes:

Bump to v0.0.6

[merge approved]


StyleCopViolations.xml output file not generated

The file "StyleCopViolations.xml" seems not to be generated anymore like in previous stylecop versions. Does it have another name?

Duplicate of #1090 / #1178.

I believe it is not needed now, as the StyleCop warnings are reported by the C# compiler as the SA*** warnings. To save build log with warnings use the /filelogger switch to msbuild (here is the documentation)

Microsoft still generates CodeAnalysisLog.xml for FxCop (which are now analyzers as well) - so I think an XML export file (in the same exact format) is still needed because there are still downstream tools in build processes that expect a StyleCopViolations.xml file...

@stgeorge At last I checked, FxCop analysis was planned for a move to analyzers but had not yet moved. I expect that the log file will disappear when the transition is complete.

There is a  Microsoft.CodeAnalysis.FxCopAnalyzers package that afaik is a port of FxCop to analyzers. Whether or not it produces an XML file I don't know, but I couldn't find any reference to it roslyn-analyzers.


Australian Scheme Feedback: (p24) Azure Example

The example configuration in Figure 13 results in a warning message being displayed due to its insecure configuration (any IP address can remotely connect to RDP). Is that the message that we want to send to readers of this document?

Replace screenshot with 'Specific IP Source Range' selected?


Better actions

This update to actions will run matrix (parallel) jobs for go 1.18 and go 1.19 with:

Fail if code is not properly formatted with gofmt
Run tests and publish artifacts
Run cover tests and publish html output

The artifacts (.html and .json) files of tests can be found at the bottom of the action's Summary page.

Is there any way we could get to see coverage, coverage improvements, etc. directly in a PR without having to manually look at the log in Actions?

@matthewsanetra Updated to use coveralls, here is our coveralls page -> https://coveralls.io/github/thecsw/gana

LGTM!


How can I add small icon logos next to experience entries?

I want to make it something like this and was wondering if that was possible with Awesome-Cv

Hello,  did you figure it out?

Nope, I'm not sure if this is possible


gio: implement FromIterator for ListStore

Fixes https://github.com/gtk-rs/gtk-rs-core/issues/951

Mind you, I have no idea what I am doing :)


Support ES module import in browser

Hi again,
I tested the library with angular and it is running into an error:
universalModuleDefinition:1 Uncaught ReferenceError: global is not defined
    at universalModuleDefinition:1:1
    at Object.9424 (universalModuleDefinition:1:1)
    at __webpack_require__ (bootstrap:19:1)
    at Module.1896 (home.component.html:14:263)
    at __webpack_require__ (bootstrap:19:1)
    at Module.721 (universalModuleDefinition:1:1)
    at __webpack_require__ (bootstrap:19:1)
    at Module.23 (app.component.html:6:8)
    at __webpack_require__ (bootstrap:19:1)
    at Module.8835 (environment.ts:16:71)

Which is expected as the angular team came to the conclusion to not support global in their webpack build for broad compatibility. https://github.com/angular/angular-cli/issues/9827#issuecomment-369578814
I think this library should not rely on some other process to supply the variable.
So going forward you could use:

globalThis: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/globalThis
polyfill it: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/globalThis#search_for_the_global_across_environments

Workaround is adding (window as any).global = window; to polyfill.ts

If it's that line, can you please test if the following will work with angular:
factory(module.exports, global || window);

It does not work, because global is not defined. You would have to check if it exists before:
var getGlobal = function () {
  if (typeof self !== 'undefined') { return self; }
  if (typeof window !== 'undefined') { return window; }
  if (typeof global !== 'undefined') { return global; }
  throw new Error('unable to locate global object');
};

factory(module.exports, getGlobal());

Or should it be global ?? window.

You can try it out in your own web browser, just open the Developer Tools and navigate to the console:

Do you have any idea how we can make an automated test for this without involving angular?

Execute the module in a browser context.
This could be a way: https://www.sitepoint.com/using-es-modules/#:~:text=Safari%2C Chrome%2C Firefox and Edge,Here's what they look like.&text=Simply add type%3D"module",executing each module only once.
Otherwise I think there is no way around implementing it inside an example project.
Which would also be a way to test it.

I'm currently using the library in react, no issues here.

I tried it out with stencil now, other error:
index-c37bab2d.js:2903 TypeError: Cannot set properties of undefined (setting 'dotnet')
    at app-home.entry.js:10:21
    at app-home.entry.js:11:3 undefined

I think I was able to reproduce this in hello world sample with the browser module as you've suggested:
<!DOCTYPE html>

Open console (Ctrl+Shift+I) to see the output.

<script type="module">
    import * as dotnet from './Project/bin/dotnet.js'

    dotnet.HelloWorld.GetHostName = () => "Browser";

    window.onload = async function () {
        await dotnet.boot();
        const guestName = dotnet.HelloWorld.GetName();
        console.log(`Welcome, ${guestName}! Enjoy your global space.`);
    };

</script>

— but using the getGlobal() doesn't seem to help:
    var getGlobal = function () {
        if (typeof self !== 'undefined') { return self; }
        if (typeof window !== 'undefined') { return window; }
        if (typeof global !== 'undefined') { return global; }
        throw new Error('unable to locate global object');
    };
    if (typeof exports === 'object' && typeof exports.nodeName !== 'string')
        factory(module.exports, getGlobal());
    else factory(root.dotnet, root);

I think this is related: https://github.com/umdjs/umd/issues/127

I've added browser-es sample for testing (not working right now): https://github.com/Elringus/DotNetJS/blob/main/Samples/HelloWorld/browser-es.html
Not sure what's the best approach to solve this (producing 2 libraries is not optimal, as it will require a lot of extra logic in C# packer). Guess the best workaround for now is polyfilling global on the consumer side. Will appreciate any suggestions.

Angular also uses Webpack under the hood.
But alright you already changed the title of the issue to reflect on the ES module issue.

Maybe angular is bundled differently, without static linking the deps? I mean, if bundle already has all the imports resolved/linked internally at build time, it shouldn't matter which import mechanism each individual dependency uses when the bundle is imported at runtime, right?

Well yes angular is already using esm modules.
Would you be open to including the polyfill for getGlobal anyway?

Would you be open to including the polyfill for getGlobal anyway?

If it'll solve the issue, sure. I've already tried doing that actually, but it didn't work, because we also need to share that global with dotnet/blazor internal stuff. But maybe I've missed something.

If it'll solve the issue, sure. I've already tried doing that actually, but it didn't work, because we also need to share that global with dotnet/blazor internal stuff. But maybe I've missed something.

It is at least solving the issue for angular, as angular seems to use a different way for importing the module.
For the import of module to work we would have to set this to window in this code of dotnet.js:
!(function (e, t) {
    "object" == typeof exports && "object" == typeof module
        ? (module.exports = t())
        : "function" == typeof define && define.amd
        ? define([], t)
        : "object" == typeof exports
        ? (exports.dotnet = t())
        : (e.dotnet = t());
})(this, () => // <== Here
...

I've tried replacing this to window, but it returned Uncaught TypeError: dotnet.HelloWorld is undefined.
Regarding angular, do you mean it's not related with the browser-es sample? In this case we'd need a minimal repro (without angular) as a starting point. Ideally, an automated test or at least a sample for manual testing.

I'm also not against dropping UMD target if there is a way to make it work with both CommonJS (require) and ES (import) in all the environments. Other UMD scenarios (global script import in browsers and AMD) are not that common nowadays, afaik.

In my browser env, no global is defined, only globalThis

I'm also not against dropping UMD target if there is a way to make it work with both CommonJS (require) and ES (import) in all the environments. Other UMD scenarios (global script import in browsers and AMD) are not that common nowadays, afaik.

If I were to do it, I would just write it all as an ES module and then polyfill it if needed.

If I were to do it, I would just write it all as an ES module and then polyfill it (like Babel) if needed.

The thing is we have autogenerated mono wasm JS wrapper and then our own runtime module. Both are bundled with webpack on top of which C# packer emits UMD initialization template. As it currently stands, we have to first figure how to configure webpack to produce a module that can be imported as both ES and CommonJS in all the environments and then update the packer initialization template.

I haven't had a chance to read your code yet, but in my experience, it can be broken down into two steps.
1, generate JS (ES6) code by template
2, depending on the configuration to let the packaging tool (Webpack) handle it (e.g. polyfill and minimization)

depending on the configuration

We have to produce a single library, that should support both ES and CommonJS scenarios.

You should allow the user to define a configuration in the C# project configuration file, such as
<JSFormat>
  esm / amd
</JSFormat>

UMD is too big and generates a lot of unnecessary code, as well as ESM would be better if the user is using a newer environment.

UMD templates are not big at all, literally about 10 lines: https://github.com/umdjs/umd/blob/master/templates
I would like to support all scenarios with a single library, if possible. After all, it's already working, except this one particular case with ES in browser.

ESM modules have been supported since NodeJS 12.
And common packaging tools (eg webpack) can escape cjs modules.

Right, but it's required in some environments, eg vscode won't work otherwise.

Right, but it's required in some environments, eg vscode web extensions won't work otherwise.

So like I said, you generate JS with the latest write-ups and then let the user choose what to do with them.

In that sense it's already working like this. You can consume current library, post-process it with a bundler (webpack/rollup) and it will work anywhere. This issue is about importing the library as ES module directly in the browser w/o any post-processing.

To check env, just:
const isBrowser =
  typeof window !== "undefined" && typeof window.document !== "undefined";

const isNode =
  typeof process !== "undefined" &&
  process.versions != null &&
  process.versions.node != null;

const isWebWorker =
  typeof self === "object" &&
  self.constructor &&
  self.constructor.name === "DedicatedWorkerGlobalScope";

const isJsDom =
  (typeof window !== "undefined" && window.name === "nodejs") ||
  (typeof navigator !== "undefined" &&
    (navigator.userAgent.includes("Node.js") ||
      navigator.userAgent.includes("jsdom")));

const isDeno = typeof Deno !== "undefined" && typeof Deno.core !== "undefined";

The "problem" with web modules (ES) is that they are incompatible with older package runtimes as they add the export keyword which may not be supported.
So, I support @Aloento view of generating an ES module first and then using rollup to create UMD/CommonJs, because generating them from ES modules is easy whereas generating ES modules from UMD is not feasible.
But in the end this is way more work than any of us should invest for this problem.
The great thing is I found this PR: https://github.com/umdjs/umd/issues/124
Which suggests the following edit:
+var getGlobal = function () {
+    if (typeof self !== "undefined") {
+        return self;
+    }
+    if (typeof window !== "undefined") {
+        return window;
+    }
+    if (typeof global !== "undefined") {
+        return global;
+    }
+    throw new Error("unable to locate global object");
+};
+var global = getGlobal();
...
-})(this, () =>
+})(typeof self !== "undefined" ? self : this, () =>

Making it work in angular, and at least loading as web module:
<script type="module">
    import * as dotnet from "./Project/bin/workaround.js";

    window.onload = async function () {
        await window.dotnet.boot();
        const guestName = window.dotnet.HelloWorld.GetName();
        console.log(`Welcome, ${guestName}! Enjoy your global space.`);
    };
</script>

But running into an error:
dotnet.js:10763 Uncaught (in promise) ReferenceError: bodyJs is not defined
    at Object.bind_method (dotnet.js:10763:49)
    at s (dotnet.js:8876:64)
    at Object.bindings_lazy_init (dotnet.js:8887:49)
    at Object.bind_static_method (dotnet.js:10849:42)
    at p (dotnet.js:13734:38)
    at boot (dotnet.js:13980:34)
    at async Object.exports.boot (dotnet.js:14138:9)
    at async window.onload (global-module.html:9:9)

Making it work in angular

Have you tried [JSFunction] stuff? I guess it won't work there as well. Mono-wasm requires a globally-accessible variable to invoke funcs in JS and looks like while this change allows routing the global into the module it's not persisting the assigned objects back.

Making it work in angular

Have you tried [JSFunction] stuff? I guess it won't work there as well. Mono-wasm requires a globally-accessible variable to invoke funcs in JS and looks like while this change allows routing the global into the module it's not persisting the assigned objects back.

Yes, I loaded your example.

Then I'm probably doing something wrong. I've tried using the getGlobal and self -> this stuff, but it still didn't work with the browser-es example. It's booting but assigned functions are not resolved by the runtime.

maybe it did not come clear from my comment:

and at least loading as web module:

which means I am running into the same errors, like body is not defined. But at least the loading of the module itself is not erroring.
I think this maybe has to with web modules effectively enforcing use strict.

Ok, so I've experimented a bit with migrating the library to es module; rewritten all the JS tests to es-style imports and switched webpack to module output: https://github.com/Elringus/DotNetJS/tree/feat/es-module/JavaScript/dotnet-runtime
I'm currently stuck on how to better wrap the webpack-exported library with the C# packer. Previously I've just copy-pasted the UMD template, but now we have to somehow wrap/override the modules exported by webpack. I'm not familiar enough with all that stuff, so any suggestions are most welcome.

webpack programmatic api is very useful
https://webpack.js.org/api/node/

This will require installing/running webpack when publishing C# project. I'd like to stick with manual patching if possible to simplify the process.

It seems manual patching is not feasible after all, given the mess wepback produce with the module target.

Will investigate AspNetCore.NodeServices.

Ah, look like it's deprecated. https://github.com/Elringus/DotNetJS/issues/53

Let me work on this.

You mean AspNetCore.NodeServices? Given it's deprecated, I don't think it's a good idea to rely on this lib.

You mean AspNetCore.NodeServices? Given it's deprecated, I don't think it's a good idea to rely on this lib.

I am now working on understanding your dotnet-runtime and modifying it for use.

PR, please check it

@Elringus I have a similar problem. I'm using React with NX (webpack) and I got the global is not defined error. I override the webpack config to add global to webpack. I now also get bodyJs is not defined.
Did you manage to get it to work in React? If so, did you configure anything special?

I'm using it with React, yes. It's a default CRA, w/o any custom building or tooling tweaks. Builds with react-scripts build.

Just tried it... it works with CRA. Had to disable ESLint for the dotnet.js file, but that's ok

@Elringus @Aloento do you know why we sometimes get the bodyJs is not defined while we do not get this using CRA?

For NX it now works if I first create a react app using CRA, and use cra-to-nx from NX.

But running into an error:
dotnet.js:10763 Uncaught (in promise) ReferenceError: bodyJs is not defined
    at Object.bind_method (dotnet.js:10763:49)
    at s (dotnet.js:8876:64)
    at Object.bindings_lazy_init (dotnet.js:8887:49)
    at Object.bind_static_method (dotnet.js:10849:42)
    at p (dotnet.js:13734:38)
    at boot (dotnet.js:13980:34)
    at async Object.exports.boot (dotnet.js:14138:9)
    at async window.onload (global-module.html:9:9)

Running into this as well for angular... is there a workaround?

But running into an error:
dotnet.js:10763 Uncaught (in promise) ReferenceError: bodyJs is not defined
    at Object.bind_method (dotnet.js:10763:49)
    at s (dotnet.js:8876:64)
    at Object.bindings_lazy_init (dotnet.js:8887:49)
    at Object.bind_static_method (dotnet.js:10849:42)
    at p (dotnet.js:13734:38)
    at boot (dotnet.js:13980:34)
    at async Object.exports.boot (dotnet.js:14138:9)
    at async window.onload (global-module.html:9:9)

Running into this as well for my angular project... any workaround?

But running into an error:
dotnet.js:10763 Uncaught (in promise) ReferenceError: bodyJs is not defined
    at Object.bind_method (dotnet.js:10763:49)
    at s (dotnet.js:8876:64)
    at Object.bindings_lazy_init (dotnet.js:8887:49)
    at Object.bind_static_method (dotnet.js:10849:42)
    at p (dotnet.js:13734:38)
    at boot (dotnet.js:13980:34)
    at async Object.exports.boot (dotnet.js:14138:9)
    at async window.onload (global-module.html:9:9)

Running into this as well for my angular project... any workaround?

Yes, try the workaround thats posted at the top the issue (inital issue description):

Workaround is adding (window as any).global = window; to polyfill.ts

Debugged this a bit more. Had to manually change the syntax in the generated dotnet.js file to get it working

Is it possible to change Angular to understand this syntax, or generate code with syntax Angular can understand?

I'm not familiar with Angular, but regarding changing it on our side: the code comes from .NET's mono-wasm; iirc, they've improved it in .NET 7, so hopefully it'll work after we upgrade (#20). I'll also look into migrating to ES bundler after that.

I'm not familiar with Angular, but regarding changing it on our side: the code comes from .NET's mono-wasm; iirc, they've improved it in .NET 7, so hopefully it'll work after we upgrade (#20). I'll also look into migrating to ES bundler after that.

It's not about Angular. It's about the type of Javascript syntax that's generated. The generated JS use implies variable declaration through expressions using commas (see result in the screenshot @tuldu attached. This syntax is much less common in JS than explicitly declare the variable with a var (as on the modified script on the right side of the screenshot).  This article discusses similar topic.
Angular doesn't parse this syntax well. I think it should be changed to the more standard and common syntax regardless Angular can parse it or not.

This is also a problem if you want to run the code in deno, since deno does not support umd.


paging view feature

Because the UIWebView inherent restraint, it can't swipe horizontal, hyphenation and text justify is also not good, can u also consider supporting these features by paging view?

Could you give me more details of what you need? is it related to issue #22?

@hebertialmeida I though I understand what @JackLauu  meaning. Like him, I would love if you can support swipe from right to left, and new page come. It looks like iBook app.

This is duplicated of #22


chore: never throw error on sites where twitter card isn't provided

close #29

Codecov Report
Attention: Patch coverage is 70.00000% with 3 lines in your changes missing coverage. Please review.

Files with missing lines
Patch %
Lines

src/transformers/link-card/link-card.ts
70.00%
2 Missing and 1 partial :warning:

Files with missing lines
Coverage Δ

src/transformers/link-card/link-card.ts
94.82% <70.00%> (-3.39%)
:arrow_down:


Duplicated images when taking photo

Setting null as default for Camera image directory. This allows the option to set a directory and avoids duplicated files when taking a photo from the image picker.

Hi @ricardosousaDev can you tell me how this can prevent the duplicated images?

@ricardosousaDev can you please elaborate how this can prevent duplicates?
Also what do you meant by "This allows the option to set a directory"

Your library is creating a public folder named "Camera" by default which is used to store the image file captured by the device's camera. What causes the duplicated images is that the system itself is saving any picture captured by the camera to the devices camera folder as well as your own folder in the external drive.
What I mean by "allows the option" is giving the option to the person who uses your library to create that extra folder if they feel like it's required.

Hi sorry for the long overdue review.
I think this is depends on the camera app?
It is not happening on the emulator and my phone. The camera app should honor the MediaStore.EXTRA_OUTPUT extra and not creating any photo on their own.
While this PR might be fix the issue that caused by "rogue" camera app, it certainly break the standalone camera function that this library provides.
I still don't know why this happen tho, even the camera basic tutorial said that we have to pass the output location


DNS Hostname for Main Orchestration Service

DNS mapping for fixed stable public ip as described in issue #57

Depends on https://github.com/eosnetworkfoundation/engineering/issues/80

resolved in https://github.com/eosnetworkfoundation/engineering/issues/80
closing as duplicate


POLIO-947: remove Responsible field from Risk Assessment tab

Remove unecesseray responsible select field in Risk Assessment tab
Related JIRA tickets : POLIO-947
Self proofreading checklist

[ ] Did I use eslint and black formatters
[ ] Is my code clear enough and well documented
[ ] Are my typescript files well typed
[ ] New translations have been added or updated if new strings have been introduced in the frontend
[ ] My migrations file are included
[ ] Are there enough tests
[ ] Documentation has been included (for new feature)

Changes

removed ResponsibleField
switch Verification score field to fix layout

How to test

Go to Campaigns
Create or edit a campaign
Go to Risk Assessment Tab: you should not see the responsible select field

Print screen / video

@cypress


Use MkdirAll when create snapshots directory

Otherwise we'll get:
mkdir /var/lib/containerd/snapshot/overlay/snapshots: file exists

when we already have snapshots there.
Signed-off-by: Qiang Huang<EMAIL_ADDRESS>

Codecov Report

Merging #692 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master     #692   +/-   ##
=======================================
  Coverage   62.79%   62.79%
=======================================
  Files           5        5
  Lines         586      586
=======================================
  Hits          368      368
  Misses        135      135
  Partials       83       83

Impacted Files
Coverage Δ

snapshot/overlay/overlay.go
60.12% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ceb159...a0dda70. Read the comment docs.

LGTM

This was actually in place to force the caller to create the root before creating the snapshotter, allowing the caller to have different perms on the parent and the child resources.
The actual bug here is that the overlay directly must be created for the snapshotter.

I think it would have also been fine just to ignore exists errors, then it would still make it explicit that this directory is created after root


Meaningful error when calling an unstubbed method with a non-nullable return type

Mockito returns null for any invocation where no expectation was set up.
If the method return type is non-nullable in Kotlin you get an NPE at a point it shouldn't really be possible given the type system's guarantees.
It would be good if you got a meaningful exception ("Unexpected invocation of method < signature > which has a non-nullable return type with arguments < args > on mock < mock >")  at the line the method was called.
package mockitotests

interface X {
  fun y(i: Int): String
}

class MockitoTest : StringSpec({

  "null safety" {

    val aMock: X = mock()

    val error = shouldThrow<Throwable> {
      aMock.y(3)
    }

    error.message shouldBe "Unexpected invocation of method fun mockitotests.X.y(kotlin.Int): kotlin.String which has non-nullable return type with arguments [3] on mock $aMock"
  }
})

Had a crack at this, but I'm not sure it's possible because mockito can't distinguish between calls to the mock passed to given or when, when the default null return is necessary, and calls to the mock in actual test execution.

I'm not sure it's possible because mockito can't distinguish between calls to the mock passed to given or when, when the default null return is necessary, and calls to the mock in actual test execution.

@Mahoney Exactly

This is an interesting case and I'm willing to leave it open to see if there's a solution to be found, although I'm afraid earlier observations may have shown this impossible.


Provide options to run ecto migrations on production

Currently, when a release is made, there is no way to call the ecto migrations on the server. Ecto could provide an option for this.
Related discussion happening on exrm: https://github.com/bitwalker/exrm/issues/67#issuecomment-80977035

This is actually already possible with Ecto.Migrator.

I can't find anything about it in the docs, where is the usage documented?

This is the module to use to run migrations manually https://hexdocs.pm/ecto/Ecto.Migrator.html.


is VulnDBGen private?

Hi,
Is your VulnDBGen repository private?
I'm doing a research on vulnerabilities and I could really use your vulnerability database or its generator.
The link to the repository is broken. Is there anyway to get it?
Thanks in advance.

I've also encountered this problem before. The link is broken. But if you click the debug branch, its readme.md has figured out steps of getting the vulnerability database.

Hi, sorry for a delayed reply. I've been on a holiday this week.
Currently, the VulnDBGen repository is private, because of some issues related to the patent application.
We plan to open it to public someday, although the time has not yet been set.
By the way, you can refer to the specific procedure for generating a vulnerability database in my paper S&P`17 paper (https://ccs.korea.ac.kr/pds/SNP17.pdf), and reproduce it. Hope it helps.


Ignore irrelevant fields in plant observation payload

If a recorded plant's species is known, the server is supposed to ignore the
value of the speciesName field in the plot observation completion API
payload, but it doesn't; instead the request fails due to a database constraint
violation.
Update the code to conditionally ignore species ID and name like the API
documentation says it does.

Current dependencies on/for this PR:

main

PR #1284   👈

This comment was auto-generated by Graphite.


Feature: Autoupdate pools.cfg

Seeing all minor changes and fixes to pools I would like to suggest to autoupdate pools.cfg file. We could download it by HTTP from github. I suggest following features:

Configure writable temporary directory "autoupdate_dir" in bh.cfg. Default is /tmp or whatever path is in windows for temporary files.
Configure refresh time for pools.cfg. Set default to 0, in case use has modified pools.cfg and we don't want it overridden
When loading configuration first check for for the pools.cfg in autoupdate_dir
md5 check file before downloading and after. In case the file changed reload config.
This could be done with new plugin system to further enhance it abilities.

Or at the very least we can print a warning if it is different from github's. I can add a plugin pretty easily.

Its added. difficulty.py autoupdates its config files and everything else will be moved over soon. However we're moving to new config file system so its a good but different solution.


Psql conflict | Made ignore parameter dependent on adapter name

As discussed in the the Issue 17, I made the ignore option available on a connection.adapter_name level.
I guess that it would be interesting to add in the future several modules for different adapters ad import the different methods according to the db adapter in use by the connection.
For now it's already an improvement, I tested my fork in an application where I am using bulk_insert and it did work like charm

@jamis is there anything I can do to help push this forward, or is this good to merge?

Hi @jamis would you fix the conflicts yourself or should I do something from my branch?

If you could address those conflicts in your branch, that would be very helpful, @mberlanda. Thanks!
You may already know very well how to do that :) but just in case, here's a link to the GitHub documentation for syncing a fork with the original repo: https://help.github.com/articles/syncing-a-fork/ . You'll do that with your branch, fix the conflicts there, and commit them. The fixes should automatically reappear here after that.

Hi @jamis ,
After the latest pull request merged with the before_save_callback functionality, I had to review my pull request a little bit. I hope that it would work fine now.
mauro

Thank you so much for your work on this! I really like how it turned out.


s6: change supervision tree permission to 0777 from 0755

this allows rootless images to make use of the supervisor
closes #240

(This is not enough to make the nginx example work as rootless, we need to give the permissions in the image some love.)

Sorry, didn't notice this was waiting on me.

Dammit the lint fails anyway :(


New version: Mozilla.Thunderbird version 115.3.1

Pull request has been created with WinGet Automation :rocket:
Microsoft Reviewers: Open in CodeFlow

Merge conflicts. Reset PR.


fix #147 by using the base path attribute instead for 2.6 compatibility.

This PR fixes issue #147. The minimum version of compatible spring-boot version has been upgraded to 2.4.0 which contains the new basePath attribute used in 2.6.0.

any chances it will ever be merged? :)


Update tests.yml

Codecov Report

Merging #122 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #122   +/-   ##
=======================================
  Coverage   57.15%   57.15%
=======================================
  Files          36       36
  Lines        1405     1405
  Branches       88       88
=======================================
  Hits          803      803
  Misses        583      583
  Partials       19       19

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a7ada3e...42099ba. Read the comment docs.


Unable to process messages when using MapValuesAsync

Description
Unable to process messages when using MapValuesAsync.
How to reproduce
StreamProcessor:
StreamBuilder builder = new();
builder.Stream<string, EventWrapper>(_appSettings.KafkaSourceTopic, new StringSerDes(), new SchemaAvroSerDes<EventWrapper>())
   .MapValuesAsync(async (message, _) => await _eventMapper.Apply(message.Value))
   .Peek((key, value) => _log.Info($"Processed message {key}"))
   .To(_appSettings.KafkaDestinationTopic, new StringSerDes(), new StringSerDes());

return builder.Build();

EventMapper:
public async Task<string> Apply(EventWrapper value)
{
    return await Task.FromResult("test");
}

No messages are processed, and no errors are logged.

For context, we've also tried setting the requestSerDes & responseSerDes parameters in the MapValuesAsync method to new RequestSerDes<string, EventWrapper>(new StringSerDes(), new SchemaAvroSerDes<EventWrapper>()) & new ResponseSerDes<string, string>(new StringSerDes(), new StringSerDes()) respectively.
It looks to me as though there's a problem in producing the initial message to the internal request topic as no messages are produced there, but the stream processor seems to get stuck when shutting down which tells me that it's busy processing something.
Is this how we should be handling producing Avro models to internal topics or is there a different way?

Hi @attributeerror  and @Numatic-Hector ,
Yes this is how you have to handle with Avro model.
Can you enable Debug logging first and post your logs in this issue please ?
var config = new StreamConfig<StringSerDes, StringSerDes>
            {
                ApplicationId = $"test-app",
                BootstrapServers = "localhost:9092",
                AutoOffsetReset = AutoOffsetReset.Earliest,
                Logger = LoggerFactory.Create(b =>
                {
                    b.SetMinimumLevel(LogLevel.Debug);
                    b.AddConsole();
                }),
                Debug = "all"
            };

Hi @LGouellec - certainly, I've uploaded the logs to Google Drive:
https://drive.google.com/file/d/15gvSclDykkCAXhVHjzTbkA7RX_WnZ3tB/view?usp=sharing

I don't see nothing special, can you start your application, publish some incoming records into the Oracle.EBS.Shipping.Events topic and attach the logs please ?

@LGouellec Certainly, uploaded to Google Drive again.
https://drive.google.com/file/d/1bintTWd0tNUDkrU9GJwOY1UgKIvK-uAg/view?usp=drive_link
Can you request access to the file please?^
I noticed this whilst all of the other messages were running past, could this be something to do with it?:
dbug: Streamiz.Kafka.Net.Kafka.Internal.KafkaLoggerAdapter[0]
      Log producer oracle-shipping-event-stream-processor-55321a20-53fc-419a-afcf-b3f374735aa2-stream-thread-0-streamiz-producer#producer-2 - [thrd:<IP_ADDRESS>:30768/2]: <IP_ADDRESS>:30768/2: oracle-shipping-event-stream-processor-KSTREAM-MAPVALUES-ASYNC-0000000005-request [0]: MessageSet with 1 message(s) (MsgId 0, BaseSeq -1) encountered error: Broker: Not enough in-sync replicas (actions Retry,MsgNotPersisted)

@attributeerror,
Yes exactly, it means the producer can't produce into the request topic because the number of in-sync replica is not enought to satisfy the acks=all.
So probably you have one or multiple broker fails, or very slow and these brokers are ejected from the ISR list.
So to resume, if the producer can't produce into the request topic, it's normal that no messages are process into the MapValuesAsync.
Best regards,

@LGouellec
Makes sense. Only problem I have is that the topic it is referencing was created as part of the MapValuesAsync topology step. How can I ensure that the topic is configured properly going forward? Could this be another issue specific to Strimzi..?
Thanks!

@attributeerror,
All internals topics are created with the same number of partitions than the source topic (in your case : Oracle.EBS.Shipping.Events) and the replication factor used is streamConfig.ReplicationFactor which by default is equal to 1.
And by default, the producer acks is configured to all.
Maybe you can list the details of the internal request topics, and see the the ISR list for each partition, and detect potential partitions under min isr. This kind of metrics is also exposed at the broker level.

@LGouellec Thanks for bringing the streamConfig.ReplicationFactor option to my attention. I hadn't noticed it before, but it's exactly what I needed. We have our cluster configured to set all topics to be created with a min.insync.replicas of 2 - which would be why we were seeing the above error. I have changed this option to default to 3 replicas and it seems to be working fine now.
Thanks for your help as always!


fix hashbang

I have php 7.1 installed, and calling ./bin/phpactor resulted in
Parse error: parse error, expecting `';'' or `'{'' in /....phpactor/vendor/dtl/class-to-file/lib/CompositeTransformer.php on line 16

which confused me for a while!
Fix this by using #!/usr/bin/env php that will use whatever php executable appears first in the user's $PATH

Ha, I was scratching my head over why the travis build was failing :)


Add support for Azure azurerm_storage_table

Add support for azurerm_storage_table
Terraform - azurerm_storage_table - https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/storage_table
Pricing - https://azure.microsoft.com/en-us/pricing/details/storage/tables/#pricing

Note for community
Please 👍 if you'd like to upvote this or receive updates.
Add a comment if you'd like to contribute.

Hi! Can I work on this one?

Can I work on this?


(needs testing)[RHCLOUD-16525] Use default image pull policy

Removes ImagePullPolicy on Deployment, Job and CronJob providers in
order to leverage defaults image pull policy.
The default policy is set based on the image tag usage:

no tag or :latest -> PullAlways
any other tag -> PullIfNotPresent

See
https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting
This improves a workflow when a deployment uses a single image tag (e.g.
latest) that is constantly being updated during development.
Ref:

https://kubernetes.io/docs/concepts/containers/images/#image-pull-policy

This needs testing. Once I get familiar with the build/test/deploy process I'll update it. Anyhow feel free to help testing this.
Thank you!

This goes against app-sre policy and was something instigated on purpose. Where are you hoping to leverage this feature?

@psav

Where are you hoping to leverage this feature?

We were developing a new service and were trying it out in EE using just the latest tag. Even if we have had a build system/scripts that would tag it with a short commit hash, it would not dismiss the fact that anyone can make a code change w/o a commit deploying it on EE. In our case we were getting some randomly cached image that was previously pulled.

This goes against app-sre policy
Do you have some reference? This does not change a regular deployments via AppSRE where a non-latest tag is used -- for those it is still PullIfNotPresent.
Alternatively we can make this configurable either in the ClowdEnv and/or within the deployment.

I did a quick search to see if this policy was documented in any App SRE repos, but I couldn't find it.  I know we had talked about it before, but that's not very helpful now.
App SRE requirements aside, I think this will require a more in-depth discussion.  We want to tightly control what kind of image tags are allowed on stage and prod for sure, but also in PR checks.
It's not hard to always push a new image tag every time you rebuild (e.g. $ uuid | cut -f 1 -d -), but I do understand the convenience of reusing latest when doing local development or rapid development in EEs.

If you all do decide to permit ImagePullPolicy to 'Always' in some cases, I just want to chime in and point out that because the Deployments do not have an ImageChangeTrigger on them (like our old DeploymentConfigs did), pushing the new 'latest' tag into the k8s docker registry won't cause the app to roll out with the newest image automatically. You'll still have to run kubectl rollout restart deployment <name>

If you all do decide to permit ImagePullPolicy to 'Always' in some cases, I just want to chime in and point out that because the Deployments do not have an ImageChangeTrigger on them (like our old DeploymentConfigs did), pushing the new 'latest' tag into the k8s docker registry won't cause the app to roll out with the newest image automatically. You'll still have to run kubectl rollout restart deployment <name>

Good to know - thanks

@bsquizz indeed, thanks! I guess we are used to restarting services when making local changes.

Superseded by https://github.com/RedHatInsights/clowder/pull/534

Correction, superseded by #536


babili issue with pino

I was using version 0.0.12 of babili to webpack my app along with the  pino module which worked perfectly - however, after upgrading babili to 0.1.1 I am now getting an error when running webpack
app dependencies:

webpack 2.6.0
babel-core 6.24.1
babel-loader 7.0.0
babel-preset-babili 0.1.1 (upgraded from 0.0.12)
babel-preset-env 1.5.1
babel-polyfill 6.23.0
pino 4.5.2

the error:
ERROR in ../~/pino/pino.js
Module build failed: Error: ~/app/node_modules/pino/pino.js: Binding not found for BindingIdentifier. onTerminatedPlease report this at https://github.com/babel/babili/issues/new
    at BindingIdentifier ~/app/node_modules/babel-plugin-minify-mangle-names/lib/index.js:216:21)
    at newFn (~/app/node_modules/babel-traverse/lib/visitors.js:318:17)
    at bfsTraverse (~/app/node_modules/babel-plugin-minify-mangle-names/lib/bfs-traverse.js:34:43)
    at Mangler.collect (~/app/node_modules/babel-plugin-minify-mangle-names/lib/index.js:260:9)
    at Mangler.run (~/app/node_modules/babel-plugin-minify-mangle-names/lib/index.js:70:14)
    at PluginPass.exit ~/app/node_modules/babel-plugin-minify-mangle-names/lib/index.js:580:19)
    at newFn ~/app/node_modules/babel-traverse/lib/visitors.js:276:21)
    at NodePath._call (~/app/node_modules/babel-traverse/lib/path/context.js:76:18)
    at NodePath.call (~/app/node_modules/babel-traverse/lib/path/context.js:48:17)
    at NodePath.visit ~/app/node_modules/babel-traverse/lib/path/context.js:117:8)
    at TraversalContext.visitQueue (~/app/node_modules/babel-traverse/lib/context.js:150:16)
    at TraversalContext.visitSingle (~/app/node_modules/babel-traverse/lib/context.js:108:19)
    at TraversalContext.visit ~/app/node_modules/babel-traverse/lib/context.js:192:19)
    at Function.traverse.node (~/app/node_modules/babel-traverse/lib/index.js:114:17)
    at traverse (~/app/node_modules/babel-traverse/lib/index.js:79:12)
    at File.transform (~/app/node_modules/babel-core/lib/transformation/file/index.js:548:35)
    at ~/app/node_modules/babel-core/lib/transformation/pipeline.js:50:19
    at File.wrap (~/app/node_modules/babel-core/lib/transformation/file/index.js:564:16)
    at Pipeline.transform (~/app/node_modules/babel-core/lib/transformation/pipeline.js:47:17)
    at transpile (~/app/node_modules/babel-loader/lib/index.js:48:20)
    at Object.module.exports (~/app/node_modules/babel-loader/lib/index.js:163:20)
 @ ./app.js 1:81-96
 @ ./index.js
 @ multi babel-polyfill ./index.js

Please could you look into why this has occurred and release a bugfix or help me to resolve the issue. many thanks.

Thanks. This is a mangler bug. I'll take a look at this.
Would be great if you can reproduce the error for a small snippet. Looks like there is some assignment/update/decl operation using the variable onTerminated and it's happening for some code like -
onTerminated = foo;
onTerminated++;
var onTerminated = bar;
// etc...

I am getting several Binding not found for BindingIdentifier errors as well.

Can you post the code where you're getting this error ?
Are you using babili in .babelrc file or are you using gulp-babili or babili-webpack-plugin ? If .babelrc are you using es2015 preset or env preset with settings that convert let/const to var

Released in 0.1.2


Put back removed ConfigGenerator.createClassName.

It will still be used for a while by older config models.
(By classes InstanceResolver and VespaModel.)

FYI: @hmusum, @aressem I think this is why at least one of the cloudconfig system tests started failing.
@hmusum, it seems that the configserver does not log when a thread dies due to not finding the ConfigGenerator.createClassName method. Probably missing some error handling. Maybe an exception is caught, but not an error.

Merging to fix broken system test.

👍


storcon: automatically clear Pause/Stop scheduling policies to enable detaches

Problem
We saw a tenant get stuck when it had been put into Pause scheduling mode to pin it to a pageserver, then it was left idle for a while and the control plane tried to detach it.
Close: https://github.com/neondatabase/neon/issues/9957
Summary of changes

When changing policy to Detached or Secondary, set the scheduling policy to Active.
Add a test that exercises this
When persisting tenant shards, set their generation_pageserver to null if the placement policy is not Attached (this enables consistency checks to work, and avoids leaving state in the DB that could be confusing/misleading in future)

Is this the behaviour we want though? We lose the shard scheduling policy whenever cplane detaches (e.g. idle tenant detach). This is also confusing.

This is perhaps a symptom of the scheduling policies being a bit over-general as a concept.  What we're using them for in practice is to use Pause to pin a tenant to a pageserver, and once we are detached, it usually makes sense to forget that pin.

When detaching or downgrading to secondary due to external location conf, the storage controller proxies it to the pageserver irrespective of the shard policy. Pageserver handles it and we preserved the policy on the storcon.

This could be fragile: it's okay if the pageserver is available, or if the pageserver is marked offline (because when it becomes available it'll get true'd up), but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant.  That's arguably tolerable, but it sort of breaks the model that we can change configs in the foreground and reconcile in the background.
I do generally agree that there's a bit of "code smell" here... but fudging the scheduling policy is probably a less bad smell than changing the config functions to do pageserver work in the foreground instead of going via reconciliation.

Added comment about usage of ShardSchedulingPolicy in 294a941f03c4b7675dae764b3065443a2d563bcb

but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant

I don't see it. We update the db and in-memory state and will reconcile when the pageserver comes back online.

But the reconcile wouldn't do anything if scheduling mode was still set to Pause (i.e. for that to work we'd have to also have the change in this PR), thinking specifically of the case of a brief unavailability where we don't do a whole-node reconcile.

Added comment about usage of ShardSchedulingPolicy in 294a941

but if the pageserver is unavailable but not yet marked offline, then it would prevent us changing the configuration of a tenant

I don't see it. We update the db and in-memory state and will reconcile when the pageserver comes back online.

But the reconcile wouldn't do anything if scheduling mode was still set to Pause (i.e. for that to work we'd have to also have the change in this PR), thinking specifically of the case of a brief unavailability where we don't do a whole-node reconcile.

Right, special casing it was implied it for detach and downgrade to secondary was implied in my proposal.


build: upgrade kafka_protocol dependency

Upgrading kafka_protocol to the latest version (4.0.3) fixes an issue with the brod_gssapi plugin: https://github.com/kafka4beam/brod_gssapi/pull/6#issuecomment-1146285067.

please update changelog.md (add to 3.16.3)

please update changelog.md (add to 3.16.3)

Fixed and force pushed.


Clear Fog::OpenStack's version cache between refresher tests

fog-openstack 1.20 caches the results of checking which API versions are supported. This upsets VCR tests that are run in sequence off the same cassette, since they expect every call to be made identically, and caching causes the version api call to be skipped. This PR clears fog's cache between tests.
@Fryguy @tzumainn

Ugh, looks like the same thing needs to happen in the network manager side.

@blomquisg hopefully travis will agree, but I think the commit I just pushed should take care of that.

Going to merge this for now as it makes master green.  Would like to investigate a more robust approach to clearing the cache after every test to ensure there won't be cross test contamination in the future.

@miq-bot add_label euwe/yes

@miq-bot add_label blocker,bug

Euwe backport details:
$ git log -1
commit 25af06f684ff102fcc22379d6e84460c1dfa8ea5
Author: Jason Frey<EMAIL_ADDRESS>Date:   Fri Mar 24 21:57:55 2017 -0400

    Merge pull request #14510 from mansam/reset-fog-version-cache-between-tests

    Clear Fog::OpenStack's version cache between refresher tests
    (cherry picked from commit 92c99b9b22e1bef2dd973e55cde0e28edbe7efe1)

    https://bugzilla.redhat.com/show_bug.cgi?id=1436340