Packed JPGs are exported as PNG with filename and mime type JPG

When using a .blend file with packed images in JPG format, they are exported as PNG (making them much bigger). Also in this case it looks like the mime-type in the glTF file is image/jpeg, which is correct, just the actual exported image files are in PNG (even though their file extension is .jpg).
Example (2.80):
FuseZombie.zip
You can use: File > External Data > Unpack All Into Files inside blender to get the expected JPGs.

I've investigated this and my conclusion is that we should get rid of a lot of the image processing code. That stuff is obviously too complex because we wouldn't have weird bugs like this if it weren't. Also I think that I have found a nice solution: Whenever we want to gather an image for the glTF, we first unpack it if it is packed. That means that we only need one algorithm that writes the image to the correct location and save us some special cases.

@UX3D-schmithuesen We would like to preserve packed data where possible.  Particularly for JPG, which is a lossy compression format, the quality will suffer with every trip through un-pack and re-packing.  Turning JPG data into PNG is particularly bad, as it pairs JPG's low quality with PNG's large size.  It's better to hold onto existing compressed image data wherever possible.

Blender's unpacking doesn't turn JPG to PNG, it just writes it to the drive. We have to write something to the drive if the user wants the textures to be separated so using Blender's own unpacking method seems much better than using ours which can only write PNG.

Maybe a bit off topic, but the exporter really should also support writing JPG when combining ORM textures. For web use the size/quality trade-off is definitely worth it in most cases.

Isn't there a way to re-use Blenders image IO? E.g. create new temp image in Blender, set content to combined ORM texture, ask Blender to save image as PNG/JPG. (Then delete image when done)

I haven't looked at how this is implemented in the this repo yet, but see this code from the old exporter:
https://github.com/KhronosGroup/glTF-Blender-Exporter/blob/992b493456abba7e4d45d78f24c8b2a48e8a5a85/scripts/addons/io_scene_gltf2/gltf2_create.py#L340-L421
In particular, note the comment on blender_image.save_render, and the fact that it unfortunately applies Scene settings like exposure to the texture output (https://github.com/KhronosGroup/glTF-Blender-Exporter/issues/184).
My only strong preference here is that if an image can be left alone, it should be left alone. If the user gives us a PNG that doesn't need to be repacked, we don't repack it. This way, at least, we preserve the option for the user to do their packing and compression in another tool. In general I think that's better than trying to intuit the correct format.

@UX3D-soares: so this will save packed JPGs as JPG?

No, this just fixes that we incorrectly call them jpg. The handling of packed files is untouched. The problem is that you can't use Blender's unpack functionality without changing the state of your scene (the image will be considered “unpacked” afterwards). We can open an enhancement issue to investigate further into this.

Hmm, I would still consider it a bug that the exporter changes the file type (I know PNG is lossless, but often size changes dramatically). How is the PNG saved, by accessing the image data and formatting to PNG in the exporter? I understand that re-encoding to JPG would loose quality but at least having the option would allow the user to make a qualified choice about size/quality. For many web use cases having textures as maybe 150KB JPGs vs. 1.5MB PNGs can make a huge difference.
Anyway, just my 2 cents :)

I couldn't manage to export 'images as planes' in .jpg format in blender, despite selecting the export option as .jpg (always exported as larger .png files). If file size is a real issue for the object I managed to just edit the glTF file in a text editor and rename the .png file as .jpg. Then drag a similarly named .jpg file into the same folder. It works!

I couldn't manage to export 'images as planes' in .jpg format in blender, despite selecting the export option as .jpg (always exported as larger .png files).

@aerialglasgow
That's because "Import Images as Planes" hooks up the alpha socket to handle images with transparency by default. When the glTF exporter sees the alpha socket is used, it thinks the texture will have an alpha channel, so that forces it to use PNG instead of JPEG.
If you turn "Use Alpha" off when you do the "Import Image as Plane" (or manually disconnect the Alpha), the glTF exporter is able to export as a JPEG.
However, it would be nice if the glTF exporter would be slightly smarter about detecting whether an image needs an alpha channel too.

Thanks @scurest Totally, it did dawn on me earlier importing another image, I had missed the 'Use Alpha' checkbox. Yes could be more intuitive on export, thanks for point this out. What an amazing app Blender is. Slowly getting to grips with it.


Array uniforms should respect SG_UNIFORMLAYOUT_NATIVE

I had a little trouble recently upgrading from sokol_gfx @ bfc41f3f13710f7fc50989215fb844a24466d9bb to aaf2c33278d047404ca434d38804a9e84bb9eda2, as the UB layouts introduced no longer match the expected size and layout for a native GLES2 shader when a float array is used (or any other sub-16-byte primitive array.) IIUC there is just an unhandled case, so I tried to handle it.

Ok merged, many thanks!
I've written a new GLFW sample/test (not committed yet) which tests float, vec2 and vec3 uniform arrays (not pushed yet). Before your PR this would already fail in the sokol-gfx validation layer (wrong uniform block size).

PS: I merged the PR with the "command line" procedure, not by pressing the PR Merge button.
Github seems to be a bit confused (maybe because your git handle "nathan" is different from the Github account "nmr8acme") and doesn't list you in the Contributors tab: https://github.com/floooh/sokol/graphs/contributors
Your commit is properly showing up in the history though (but note how your name isn't clickable):
https://github.com/floooh/sokol/commits/master
Just in case you're wondering what's up with that ;)

Link to sample: https://github.com/floooh/sokol-samples/blob/master/glfw/uniformarrays-glfw.c

And Changelog update (15-Jan-2022): https://github.com/floooh/sokol/blob/master/CHANGELOG.md

Thanks :)


Move storage/auth/auditLogger/theme into launcher

Part of #99
Description
This moves the following repos into this one:

@flowforge/nr-audit-logger
@flowforge/nr-auth
@flowforge/nr-storage
@flowforge/nr-theme

I have test this locally, but experience tells me the paths around module loading at subtly different in docker/k8s land. Getting this raised now to get some eyes on it, but will want to finalise some docker-based testing before we merge.
Checklist

[x] I have read the contribution guidelines
[-] Suitable unit/system level tests have been added and they pass
[ ] Documentation has been updated

[ ] Upgrade instructions
[ ] Configuration details
[ ] Concepts

[ ] Changes flowforge.yml?

[ ] Issue/PR raised on flowforge/helm to update ConfigMap Template
[ ] Issue/PR raised on flowforge/CloudProject to update values for Staging/Production

Labels

[ ] Backport needed? -> add the backport label
[ ] Includes a DB migration? -> add the area:migration label

Should the icons be included twice?

Possibly not. They are in both places in the original repo, but I can't immediately see any use of them from the common folder.
May need @Steve-Mcl to confirm, but I think we can remove the images from lib/theme/common as they only ever get loaded from resources/

I have a vague memory of changing things around to use the resources but may not have "clean up" afterwards.
The idea was (at the time) that the theme can have own icon or fall back to common but I am unsure where it got to in the end. (sorry)

OK, I'm going to test this in a container this afternoon, assuming no other issues I'll test with/without the extra images and then merge as appropriate

OK, so I removed the nr-theme line mention in the package.json dependencies and tested.
Auth, Storage and Audit logging appear to be working, but the theme has not been loaded.

Team libraries are also not working

I think I've fixed it

Okay - so I think the only remaining task is to remove the duplicate icons. I've test that locally and if it looks good, will push then get this merged.


InputUnstyled with 'multiline' property logs warnings for 'maxRows' and 'minRows'

Duplicates

[X] I have searched the existing issues

Latest version

[X] I have tested the latest version

Current behavior 😯
When having InputUnstyled with 'multiline' logs warnings  for 'maxRows' and 'minRows':
Warning: React does not recognize the 'minRows' prop on a DOM element. If you intentionally want it to appear in the DOM as a custom attribute, spell it as lowercase 'minrows' instead. If you accidentally passed it from a parent component, remove it from the DOM element.

Warning: React does not recognize the maxRows' prop on a DOM element. If you intentionally want it to appear in the DOM as a custom attribute, spell it as lowercase 'maxrows' instead. If you accidentally passed it from a parent component, remove it from the DOM element.

Expected behavior 🤔
Should not logs an warnings.
Steps to reproduce 🕹
Steps:

Have a InputUnstyled with 'multiline'
Check console

https://codesandbox.io/s/unstyledinput-material-demo-forked-9nqyj
Context 🔦
I think that we need to delete this props (maxRows and minRows) which is not DOM native before we spread them in Input.
      <Input {...inputProps} />
Your environment 🌎

`npx @mui/envinfo`
  Don't forget to mention which browser you used.
  Output from `npx @mui/envinfo` goes here.

+1, I'm seeing this too, and it's put us in a pickle: until we upgrade to 5.x one day, it seems that our choice is either to see console errors because (1) we should be using minRows instead of rows in 4.11.x, or because (2) using minRows causes that prop to be improperly passed down to <input> in 4.12.3, which appears to be the most recent 4.x release.
Would it be possible to get a quick fix for this on the 4.x branch so that we can resolve our console errors related to TextField rows once and for all, or is the recommendation to just bit the bullet and upgrade to 5.x? (Fingers crossed that the former is possible.)


Flag that generates explanation for Vyxal code.

It's nice when computers can do stuff for you.

It's not nice when computers can spam you.

The only problem I have with automatically generated explanations is that they aren't helpful for understanding how a program works. Sure an auto generated explanation can tell you what each individual component of a program does, but it doesn't tell you how the components interact or any other logic. Also, it doesn't allow for easy semantic grouping (like how some people might group 2/* together as "half then multiply").
Maybe a flag that places the # at the end of each line would be a compromise - users could define semantic blocks that would be grouped on the same line with the explanation syntax already formatted for them. I think that would be a good and useful idea.

Secretly uses ChatGPT in compiler code to explain what it does

I think it would still be helpful, even without full semantic grouping. But we can detect, for example, a number/string literal next to a two/three-argument element.

https://github.com/Vyxal/Luminespire
Rather than a flag, I've decided to make an external tool that:
a) can be customised to have special tools for vyxal and
b) is otherwise language agnostic and can be used by the greater code golf community

So does that this mean this PR is going to be closed?

I'm closing this as not planned for now because we have Lumenspire. If lyxal decides this is a good idea in the future, we can reopen it.

I would say this is completed with the introduction of Luminespire.

Lumenspire isn't a flag that automatically generates an explanation, so this hasn't been completed.

Lumenspire isn't a flag that automatically generates an explanation, so this hasn't been completed.

@nayakrujul but it has been completed. I view it as completed.

What? You think Luminespire is a "flag that generates explanations for Vyxal code"?

What? You think Luminespire is a "flag that generates explanations for Vyxal code"?

Well actually Luminespire was intended to close this issue.


ERROR in NguiUtilsModule is not an NgModule

I'm trying to import NguiUtils to my app but got this error.
ERROR in NguiUtilsModule is not an NgModule
In app,module.ts
import { NguiUtilsModule } from '@ngui/utils';
@NgModule({
imports: [
....
NguiUtilsModule
....
Installed version is
"@ngui/utils": "^0.8.1",

As a note to those that find this when they have the same problem, deleting the
import { NguiUtilsModule } from '@ngui/utils';
line from your app.module  (or whichever module it's defined in), building, then putting the line back, seems to remove this error (until next time!)

This issue isn't fixed yet. Please fix it.


Single Tap on TextField does not display Toolbar

Internal: b/173627308
During final review of an app in the process of being launched, we found a surprising issue: If I, the developer, want to paste into a TextField, I long-press the field, and the toolbar appears. If the producer wants to paste into a TextField, they tap the already-focused Field...and nothing happens.
On every other app we could find, that tap presents the toolbar. As far as I can see from the documentation, Flutter's code, pub.dev, StackOverflow, and Google, there is no current issue tracking this behaviour, method providing this behaviour, or explanation of how to do so.
What I'm looking for is the ability to react to a tap on an already-focused TextField, and present the toolbar (e.g. "Paste", "Select All").

Is there a way to do this currently that I missed? (If so, great! It's entirely likely I just failed to find it, and it could even be "obvious".)
If not, what is the best way to go about adding that? We've decided upon Flutter as our platform of choice, and I'd like to start contributing as we transition from "launch our first app" to "maintain all our apps".

Hi @Schoonology
can you please provide your flutter doctor -v ?
Thank you

Sure.
[✓] Flutter (Channel stable, v1.12.13+hotfix.5, on Mac OS X 10.14.6 18G95, locale en-US)
    • Flutter version 1.12.13+hotfix.5 at /Users/schoon/Code/community/flutter
    • Framework revision 27321ebbad (4 weeks ago), 2019-12-10 18:15:01 -0800
    • Engine revision 2994f7e1e6
    • Dart version 2.7.0

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/schoon/Library/Android/sdk
    • Android NDK location not configured (optional; useful for native profiling support)
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.3, Build version 11C29
    • CocoaPods version 1.8.3

[✓] Android Studio (version 3.5)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 40.2.2
    • Dart plugin version 191.8593
    • Java version OpenJDK Runtime Environment (build 1.8.0_202-release-1483-b49-5587405)

[✓] VS Code (version 1.41.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.7.1

[✓] Connected device (1 available)
    • iPhone 8 • 4EE78652-B1E5-4F6E-9671-95CB1F995FC3 • ios • com.apple.CoreSimulator.SimRuntime.iOS-13-3 (simulator)

• No issues found!

CC @justinmc

I do not quite understand the issue. The current implementation that show toolbars upon long press and double tap. Single tap does not bring up toolbar. If you are looking for reacting to single tap, does this work for you? https://api.flutter.dev/flutter/material/TextField/onTap.html

Native iOS textfields do appear to (somewhat unreliably) show the text selection toolbar when an already-focused text field is tapped. If that's what iOS users expect we should probably implement it; we haven't done so yet though.
CC @LongCatIsLooong

I tried checking which gestures show the selection menu on each platform (on physical devices).  A ✅ means that it shows the menu, and a ❌ means it does not show.  In all cases, I had something copied to the clipboard.

Gesture
iOS
Flutter on iOS
Android
Flutter on Android

tap
✅
❌
❌
❌

double tap
✅
✅
❌
✅

long press
✅
✅
✅
✅

It seems like Flutter should match each platform's behavior (regardless of whether TextField or CupertinoTextField is used).  @Schoonology would that solve your problem?

@justinmc It seems like that will completely solve my technical problem.
I'm happy to move this conversation somewhere else, but I'd like to use this as an opportunity to start contributing to Flutter. (I've done this with other platforms, but am not set up to do so here.) What does that require? Is this a good issue to start with?

@Schoonology That's great!  This is not a bad issue to start with, and I'm happy to help.
Start by looking at the contributor guide, especially "setting up your framework development evironment".
The part of the source code that you'll want to start looking at is the calls to showToolbar in text_selection.dart (example).  Notice that TextSelection is also overridden in material/text_selection.dart and cupertino/text_selection.dart.
Feel free to reach out to me if you need any pointers.  I'm usually online on the Flutter discord channel (https://discordapp.com/invite/BS8KZyg) and I think my email is in my Github profile.

Same happens with CupertinoTextField

@chunhtai
You brought up reacting to onTap events as a possible solution. How do you open the toolbar manually when a specific event happens?

@CiriousJoker If you want to bring up the built-in tool bar, I don't think it is possible with current API. The problem is you don't have access to the underneath EditableText to call showToolBar because the TextFieldState is private class(Although you can manually patch it). You will have to use editable text directly, and it is very cumbersome. It is probably make more sense to wait for the fix.

What exactly does the fix include? Does it directly expose the showToolbar() function or does it patch the default behavior?
I specifically want to open it manually, I don't care about the default behavior at all.
I want to show the paste controls when tapping on the empty text field and select everything + copy/paste when tapping when there's existing content.

According to previous comments, I think the plan is to patch the default behavior. @justinmc might be able to confirm.
For your use case, I would probably use editable text directly.

I hope to fix the show conditions in my comment https://github.com/flutter/flutter/issues/48434#issuecomment-572851932 by calling to showToolbar when the corresponding event happens.  I don't plan to directly expose showToolbar on TextField.  @CiriousJoker could you open a separate issue about being able to programmatically show the text selection menu?

flutter doctor -v
[✓] Flutter (Channel dev, 1.20.0-2.0.pre, on Mac OS X 10.15.5 19F101, locale
    en-GB)
    • Flutter version 1.20.0-2.0.pre at /Users/tahatesser/Code/flutter_dev
    • Framework revision 15a28159bc (10 days ago), 2020-06-23 04:52:58 -0700
    • Engine revision 91a63d6a44
    • Dart version 2.9.0 (build 2.9.0-19.0.dev 7e72c9ae7e)

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
    • Android SDK at /Users/tahatesser/Code/sdk
    • Platform android-29, build-tools 29.0.3
    • ANDROID_HOME = /Users/tahatesser/Code/sdk
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.5)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.5, Build version 11E608c
    • CocoaPods version 1.9.3

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 47.1.2
    • Dart plugin version 193.7361
    • Java version OpenJDK Runtime Environment (build
      1.8.0_242-release-1644-b3-6222593)

[✓] VS Code (version 1.46.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.12.1

[✓] Connected device (5 available)
    • SM M305F      • 32003c30dc19668f          • android-arm64  • Android 10
      (API 29)
    • Taha’s iPhone • 00008020-001059882212002E • ios            • iOS 13.5.1
    • macOS desktop • macos                     • darwin-x64     • Mac OS X
      10.15.5 19F101
    • Web Server    • web-server                • web-javascript • Flutter Tools
    • Chrome        • chrome                    • web-javascript • Google Chrome
      83.0.4103.116

• No issues found!

I put some effort in trying to write one-off fixes for a few of the inconsistencies that I called out in my comment above in a branch, but I think it's going to take a bigger refactor to get all of this right.  The code is getting a bit convoluted when several different files need to be able to decide when the text selection menu shows or does not show.

@justinmc That's good to hear. I took a few cracks at making this work, too, but had a lot of difficulty getting the desired behaviour without causing other issues.

Yeah exactly.  There's no clear place where you can write something like "for Material text selection, when a double tap happens, do this".  The logic is spread between the 3 text_selection.dart files plus EditableText and TextField/CupertinoTextField.  I think we need to refactor it so that material/text_selection.dart and cupertino/text_selection.dart are able to have full control over deciding when their text selection menu shows up.

I've found the place but it's too complicated to make PR to Flutter right. Maybe I'm wrong.
The place is     https://github.com/flutter/flutter/blob/804c3ff31f6e517799bdd262c6b6cbe6eafac793/packages/flutter/lib/src/material/text_field.dart#L105
you need add after:
if(_state._effectiveFocusNode.hasPrimaryFocus && delegate.selectionEnabled && shouldShowSelectionToolbar) {
    editableText.showToolbar();
}

I'll appreciate if someone with all fits push this fix.

If I make that exact change, the toolbar doesn't hide with subsequent taps.  Even if I change it to .toggleToolbar().  Maybe there's some smallish change that will work, but really this needs a deeper refactor.

@justinmc My working solution looks like now:
   if (delegate.selectionEnabled) {
      switch (Theme.of(_state.context).platform) {
        case TargetPlatform.iOS:
        case TargetPlatform.macOS:
          switch (details.kind) {
            case PointerDeviceKind.mouse:
            case PointerDeviceKind.stylus:
            case PointerDeviceKind.invertedStylus:
              // Precise devices should place the cursor at a precise position.
              renderEditable.selectPosition(cause: SelectionChangedCause.tap);
              break;
            case PointerDeviceKind.touch:
            case PointerDeviceKind.unknown:
              // On macOS/iOS/iPadOS a touch tap places the cursor at the edge
              // of the word.
              renderEditable.selectWordEdge(cause: SelectionChangedCause.tap);

              if (_state._effectiveFocusNode.hasPrimaryFocus &&
                  delegate.selectionEnabled &&
                  shouldShowSelectionToolbar) {
                editableText.showToolbar();
              }
              break;
          }
          break;

Why this is closed?
The problem exists.
All duplicates are closed too.

Why this is closed?
The problem exists.
All duplicates are closed too.

This issue is currently open as it should be.
I'm currently working on a refactor that should alleviate some of the problems with the codebase that I mentioned above.  After that I think we'll be able to address the inconsistencies from my table above.

This issue is currently open as it should be.
I'm currently working on a refactor that should alleviate some of the problems with the codebase that I mentioned above.  After that I think we'll be able to address the inconsistencies from my table above.

@justinmc any notable update on this? I'd like to hear any progress you've been able to make. I believe iOS users are still experiencing some friction interacting with text fields because of this bug. Thank you for your efforts so far!

This is the PR that will unblock the clean fix of this and other bugs: https://github.com/flutter/flutter/pull/84232 CC @LongCatIsLooong
Once we are able to handle tap interactions via Actions, then I think we'll be able to untangle problems like this by creating one place where the the logic for all platforms lives.

Any updates on this?

We are in the middle of refactoring so that we can handle things like this using the Actions system.  It should make this much cleaner and less prone to errors.
It could be awhile until that is done, though.  If anyone wants to write an interim fix in the meantime feel free.

Is there any sort of workaround to get a single tap to bring up the toolbar on iOS? Using CupertinoTextField doesn't seem to work.
My app's primary UI flow involves pasting text into a field. It's in beta, and multiple testers have already reported that they can't paste by tapping on the field. I'm worried about releasing to production when the primary interaction feels wrong on iOS.
Any thoughts on how I can work around this until the real fix is possible?
(cc @justinmc)

No workaround that I'm aware of, unfortunately.  I'm open to reviewing a PR for this if anyone vitally needs this without waiting for the refactor.  We're making progress on the refactor but it's a big change.

I created a PR for this over at #95644 if you're willing to take a look at some point @justinmc. :) It's my first contribution and I'm not sure if I did it right though! 🤞

@justinmc
Can we have onToolbarOptionChange callback? This will help us To detect the changes of the toolbar option.
Imagine that we have custom autocomplete, and we want to disappear our overlay when the toolbar has been opened.
This is what exactly we need in our app.

On my Android (Samsung S21) device the toolbar is shown after a single tap on native apps when the text field is already focused.

Is there any sort of workaround to get a single tap to bring up the toolbar on iOS? Using CupertinoTextField doesn't seem to work.
My app's primary UI flow involves pasting text into a field. It's in beta, and multiple testers have already reported that they can't paste by tapping on the field. I'm worried about releasing to production when the primary interaction feels wrong on iOS.
Any thoughts on how I can work around this until the real fix is possible? (cc @justinmc)

Did you end up finding a fix for this?

This was fixed by #111995.

Please help me, maybe I'm doing something wrong.
I switched to the master channel to test opening a toolbar with a single tap, but it didn't work for me both on Android and iOS. The toolbar still only opens with a double tap or long press.


Carthage install not working

Here's my cartfile
github "pepibumur/sugarrecord"
This is what I get back
computerName:AppName user$ carthage update
*** Fetching sugarrecord
*** Fetching Result
Parse error: expected submodule commit SHA in output of task (ls-tree -z HEAD Library/ObjectMapper) but encountered:
computerName:AppName user$
I'm new to Carthage but is something wrong with one of the submodules?

I am also unable to install via Carthage. I just get *** Fetching SugarRecord and then it hangs forever.

Thanks for reporting it @jodyheavener and @Biclops .
I'll have a look

@jodyheavener I suppose the SugarRecord project is too large.
git clone
Receiving objects:  30% (3169/10364), 39.18 MiB | 39.00 KiB/s
It's about 122.81M, it should takes much time to download it.
My solution is git clone it firstly, and then use a local project in Cartfile

Carthage support has been disabled in the last version.


Dragon Ball Random Defense 8.2

Hey,
Im trying to deprotect Dragon Ball Random Defense 8.2 but it's not working somehow.
There are aboout 40 missing files after the scan which I should detect ingame while playing the map and i've done almost everything possible but im unable to find the missing files.
I want to update the Map myself with adding new Characters, Functions etc. but im unable to deprotect it.
Map - Link: https://maps.w3reforged.com/maps/categories/hero-defense-and-survival/PG DRD

The 40 missing files is normal. It typically means it's a 3d model that was imported by the map creator but they never assigned it to a unit, so it's not actually needed. If you actually playtest the deprotected map & it's missing a 3d model, then you'll need to fix it yourself, but this is rare.
The biggest issue is that the war3map.j is not deprotecting into the triggers. This is strange, I haven't seen that before. I'll have to research to fix it. I'm a little busy with other projects right now, so I can't promise to get to it very quickly.
If you know c#, you can try to fix it yourself, all the source code is in this github repository.
Otherwise, you'll need to fix the map yourself using traditional manual deprotection techniques, like MPQEditor.
Or, you can always move on to a different map.

Hey,
Thanks for the reply. Well im unable to play the the deprotected Map at all,
also if i try to open it in the editor it always says they're some missing triggers.

I've just released a new version of my tool, please try it out and see if it works better for you.
I can quick general advice over discord, my username is speige
I generally don't have time for custom requests, but if you want me to actually fix the problems for you, I can do it for a fee. My donation page is https://github.com/sponsors/speige

I've just released a new version of my tool, please try it out and see if it works better for you.
I can quick general advice over discord, my username is speige
I generally don't have time for custom requests, but if you want me to actually fix the problems for you, I can do it for a fee. My donation page is https://github.com/sponsors/speige


Update NMFS pkgdown template

This entails editing the NOAA themed pkgdown blog post.
Here is the pkgdown template repo

We moved this repo over to NMFS Open Sci (Eli, Bai and Christine all agreed this was a good place to put it). And Kathryn and I edited the pkgdown blog post on the FIT website. My next step will be to test the instructions with an example R package, and then I think this task will be complete!

I tested the tutorial with an experimental package of my own (palettehelper) and it worked great! I started with a pre-existing package repo, as we discussed.
So the final assessment of this was; we moved the repo over to NMFS Open Sci, and revised the pkgdown blog post for the FIT blog. Going to close this issue because I think we are finished!

@MargaretSiple-NOAA  I was mostly able to follow the blog to add pkgdown to someone's site with the following notes

I had to already know what to do re GitHub Action to follow those instructions.

The footer image is huge for some reason.

Also there is some weirdness in the css. Look at the side navbar

Thanks Eli, and yikes re: that icon size! That's a drive-thru icon. I did some experimenting with adding a hex sticker and can add that to the instructions. For a little more elaboration on the GitHub Actions part, did you use the {usethis} way, or did you use it without {usethis}?
I'll see what's up with the css. I just noticed that too. I hadn't looked at the references page yet. I think I just inherited the css from the previous version so may have to do some digging.

I put in a PR to fix the image size. I'd forgotten that I'd fixed this some time ago in my pkgdown template but that fix didn't get migrated upward.

Re logo:

Add logo to man/figures/logo.png
Then add the top of your README.md like so

# yourpackagename <img src="man/figures/logo.png" align="right" />

pkgdown will handle the rest including creating a favicon for you. Just make sure to call your logo logo.png

Re GitHub Action, I used the gh4actions package but I knew what I was doing. It would be good to watch someone try to apply the instructions if they have no or limited experience with GH Actions.

Thanks Eli, and yikes re: that icon size! That's a drive-thru icon. I did some experimenting with adding a hex sticker and can add that to the instructions. For a little more elaboration on the GitHub Actions part, did you use the {usethis} way, or did you use it without {usethis}?
I'll see what's up with the css. I just noticed that too. I hadn't looked at the references page yet. I think I just inherited the css from the previous version so may have to do some digging.

@MargaretSiple-NOAA do you think we still need to make edits to the blog post? If so, feel free to put coworking time on my calendar so we can finish it up!
@eeholmes , if you have thoughts on additional documentation need for ghactions, please feel free to make changes on the blog post and submit it as a PR. I cannot tell what is missing, but if someone from the FIT committee with limited ghactions experience could volunteer to test out the blog post, I think that could be helpful!

Re GA documentation, Let's ask someone inexperienced try to follow the instructions at the next FIT meeting.

So sorry for the delay on this you two. I plan to do this work this week. We will likely need to edit the blog post but I'll take a look and see what we should revise. I had no trouble following the instructions and don't frequently use GHA, but I can revisit.

@Bai-Li-NOAA just fixed the CSS issue here: https://github.com/nmfs-fish-tools/nmfspalette/pull/31
I think it should auto update if people are refering to the nmfspalette css file.
As for gha, I think it is fine to leave it as-is if @MargaretSiple-NOAA is not a frequent GHA user and found the instructions possible to follow. I'm going to close this, but feel free to reopen if there are things to revisit.

Thank you Kathryn! Yes if the css change fixed the menu issue, that's probably the most glaring issue. I am not a frequent GHA user and was able to follow the instructions, but can run through them again for thoroughness. If I find anything crazy, I'll reopen but I think they're okay.
I have a fork where I'm adding instructions for adding a logo / favicon and will make a pull request.

@MargaretSiple-NOAA yes, it will solve the menu issue.
Thanks for working on the instructions for adding a logo! I think it's ok to leave the github actions instructions as-is.


Enable API URL

In new version of IntelliJ IDEA 14.1 is support for Docker. For use it is important to enable API URL.
What about this for Kitematic?

Thanks for the issue.
To help me understand this better, can you point to an IntelliJ tutorial or similar? Are you looking for how to get the Docker Remote API URL that Kitematic uses?

I think this is what @DaspawnW mention - http://blog.jetbrains.com/idea/2015/03/docker-support-in-intellij-idea-14-1/

Ah! Ok. Thanks @Jirapong

So, I believe that if forward the Remote API URL port 2375 or 2376 on the "dev" VM manually it should work?
May be this is good to be an option on the VM Settings?

Thank you, @Jirapong. I think this is what I mean.
Sorry for my late answer.

Wow! Thanks for the help @Jirapong !
Intellij doesn't work with an engine URL over TLS?

I believe they do also. However, On OSX there is issue with curl library that not allow --cert and --key. see reference at https://github.com/boot2docker/boot2docker/issues/573 . I'm working on the solution to allow TLS.

OK. On connecting with TLS. I put everything in my blog post at https://github.com/kitematic/kitematic/issues/348

@Jirapong awesome. Thanks for the blog post!

@Jirapong  Thanks for the blog.
Assuming the information solves the issue. Closing this issue for now. Please re-open if you feel the need to. :)


AWS

[X] Setup RDS
[ ] Setup Container things
[ ] Setup Auto Scailing

RDS Connection string is sent in the Telegram Chat.
DATABASE_URL=postgres://username:password@localhost:5432/<dbname>
the  can be different depending on the service. But likely gonna just use one so that they are one linked schema instead of separated things


tests/ envs

I've added a GDBPATH env for specifying which GDB executable to use in multihop-hand and also noticed an unused BABELPATH variable; made all the envs overrideable, e.g.:
$ sudo GDBPATH=cgdb ./multihop-basic.sh # Someone might want to use a GDB wrapper
$ sudo BABELPATH=some_other_babeld_exec ./multihop-basic.sh # Someone might want to test a different babeld exec for some elusive bug/performance drop/whatever without running around different repos
$ sudo LABPATH=~/other_lab_script.sh ./multihop-basic.sh # ...same goes for network-lab.sh scripts

03d4a09 - I've also enabled the local config server where it makes sense and turned off optimization so that debugging can be easier.

looks good, lets merge.


form-group labels overlap other objects on the page

Select one ...  (check one with "x")
[x] bug
[ ] feature request
[ ] enhancement

Expected behavior
The div.form-group element should resize according to the nested input and label elements
Actual behavior
The div.form-group element resizes according to the nested input only
Reproduction of behavior

Include a link to the reproduction scenario you created by forking one of the Clarity Plunker Templates:

https://plnkr.co/edit/ilCAllBWbTxzn16MgAkO?p=preview
Environment details

Angular version: None

Clarity version: Latest - 0.9.4

OS and version: OS X El Capitan

Browser: [Chrome XX | Firefox XX | IE XX | Safari XX]

Hey @hristovh: We already have this issue raised (see here: https://github.com/vmware/clarity/issues/626) and we are looking into this. The fix that we have identified might be a breaking change and we are evaluating the impact of it. For now the best way to deal with forms which might have wrapping labels is to use forms inside of grids (https://vmware.github.io/clarity/documentation/forms - Refer to the "Forms in Grid" sections). Closing this issue as a duplicate.

Hey there @hristovh I took a look at your plunker.
wrt your markup structure: .form-group and .alert .alert-warning are different in that form-group class is for layout and alert* classes are for presentation.
Take a look here where I modified your markup: https://plnkr.co/edit/tI05y6ueqEKA06fK2vL7?p=preview
While it doesn't fix the underlying issues with forms it might help you mitigate it until we address that.


get_timber_image_responsive_acf doesn't deliver image but error message

Hi,
if I use the function get_timber_image_responsive_acf in my twig files like this:

I get following notice:
Notice: Trying to access array offset on value of type int in \mindkomm\timmy\functions-images.php on line 452
src=“ sizes="(min-width: 43rem) 860px, 100vw" srcset="//localhost:3000/app/uploads/2020/04/img-375×0-c-default.jpg 375w, //localhost:3000/app/uploads/2020/04/img-500×0-c-default.jpg 500w, //localhost:3000/app/uploads/2020/04/img-750×0-c-default.jpg 750w, //localhost:3000/app/uploads/2020/04/img-800×0-c-default.jpg 800w, //localhost:3000/app/uploads/2020/04/img-860×0-c-default.jpg 860w, //localhost:3000/app/uploads/2020/04/img-1000×0-c-default.jpg 1000w, //localhost:3000/app/uploads/2020/04/img-1125×0-c-default.jpg 1125w, //localhost:3000/app/uploads/2020/04/img-1500×0-c-default.jpg 1500w, //localhost:3000/app/uploads/2020/04/img-1800×0-c-default.jpg 1800w, //localhost:3000/app/uploads/2020/04/img-2100×0-c-default.jpg 2100w, //localhost:3000/app/uploads/2020/04/img-2400×0-c-default.jpg 2400w"
src="data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7"“ alt=““ title=““>

The image does not show, the HTML looks like this: <img class=""

When using this instead:  it works perfectly.
Is it just me or did anyone encounter this problem as well?

Nah, it’s not only you, it‘s probably everybody 😊. The reason is that this was a function I added very early when developing Timmy, and I didn’t touch it for a long long time, because there wasn’t a use case I would use it on.
I added a fix in https://github.com/mindkomm/timmy/commit/ac9871a4dc01cce6590aa210d13e21f0ae6c5f06 together with tests, so that errors like these shoudln’t happen in the future for the get_timber_image_responsive_acf() function.
This should now be fixed with the 0.14.5 release.


Replace all the various (-want, +got) styles with a single format

Changes
Many, many of our tests use cmp.Diff to test whether two objects
match. Every time that func gets used the test-writer has to remember
to include "(-want, +got)" in the string they print. This is annoying
to remember the order and format and can be easily implemented in a
helper func instead.
The first commit in this PR introduces a helper func to print diff messages in a
consistent way. This means test writers dont have to manually type
this formatting out every single time. The func is called diff.PrintWantGot().

Submitter Checklist
These are the criteria that every PR should meet, please check them off as you
review them:

[x] Commit messages follow commit message best practices

See the contribution guide for more details.

AMAZING

/approve

wow, this is a huge cleanup, its always confusing to remind myself which one means want + or - 🤔. This will help a lot, thanks @sbwsg 👍
Build is failing on imports order 😢 :
-	"github.com/tektoncd/pipeline/test/names"
 	"github.com/tektoncd/pipeline/test/diff"
+	"github.com/tektoncd/pipeline/test/names"

/lgtm

Thanks for reviewing @pritidesai !

Build is failing on imports order 😢

Thank goodness for the Makefile @vdemeester introduced, I ran make goimports and it fixed them all (I hope!)

/test pull-tekton-pipeline-integration-tests

Can we have a presubmit check to ensure I don't add a new cmp.Diff call that bypasses the helper? Otherwise I'm going to make sure we're going to need one of these PRs every few quarters 😅.

Can we have a presubmit check to ensure I don't add a new cmp.Diff call that bypasses the helper? Otherwise I'm going to make sure we're going to need one of these PRs every few quarters sweat_smile.

Let's do that in a follow-up :wink: https://github.com/tektoncd/pipeline/issues/2594


When some task push very big object to other party, it wll block others

To reproduce in fate:
1. Task 1 push(remote) very big object to other site.
2. Submit another job, create job will be failed.
Features we need:
1. single push can not use full source.
2. command channel and task data push channel maybe better to be separate

This seems to be a grpc feature. We are looking into it and see if there is anything we can do.

Hi @mgqa34 , thanks for the feedback. But we have some questions upon your descriptions:

How big is the object and what is the type of this object?
Can you elaborate your use case?
Please elaborate the features you need and their use cases, and why these features help.


Consider switcing to xz/lzma compression for release archives

Hi,
New dist file with generated documentation (-dist) and configure script is now over 6Mbyte in size (1.47.0) which is quite a jump from 1.46.0. Switching to xz/lzma would cut this is in more than half, downside is that it takes slightly longer to generate but given it's a small library I think that's worth the tradeoff.

I fixed the current files and re-released those.
It seems the trouble is the way we use EXTRA_DIST
https://www.gnu.org/software/automake/manual/html_node/Basics-of-Distribution.html
https://github.com/libuv/libuv/blob/815693f71523175f577e3cbdb9cf576d30ea9250/Makefile.am#L121-L131
I am not sure of a good way around that, since we cannot use git ls-tree in the dist tree (the git files are not copied) and we cannot even use $(wildcard) or find since automake will error with a warning it is non-portable.

Thanks!


chore: revert async aggregate with randomness

Motivation
Change is choking network thread when —subscribeAllSubnets flag is applied.  Need to further investigate that issue so reverting for 1.23 and will reapply for 1.24 once network issue is resolved.

this reverts #7204
for more info, the above PR seems to improve main thread a lot that it caused low performant on network thread due to high gc
this is peer count:

there are too few inbound peers while a lot of out bound peers, while mesh peers are a lot which we haven't seen in past releases

gc is so bad on network thread

:tada: This PR is included in v1.23.0 :tada:

:tada: This PR is included in v1.24.0 :tada:


fix: absolute path moduleNameMapper + jest.mock issue

Summary
In this special case, the moduleNameMapper got applied to the import path twice, resulting in an invalid module path.
still validating the changes, WIP
Fixes #8633
Test plan
Added the absolute path moduleNameMapper test case.

Any news here? It seems sane to me, and tests are passing. Rebasing on master should fix the CI failure on circus

@chauchakching ping 🙂. Is it ready to be out of Draft now?

Almost forgot I had this PR...
Rebasing

Thanks @chauchakching! It seems tests are failing on CI, would you be able to take a look?

Thanks @chauchakching! It seems tests are failing on CI, would you be able to take a look?

of course

2 hg tests failed on my mac though, even on master branch

Yeah, ignore the mercurial tests, they only work on very specific versions of mercurial

Thanks!


About Win32 default icons(关于win32默认图标)

我想知道这个字符串是干嘛用的？
I want to know what this string is for?

On the Windows platform you can specify an icon resource in your executable called "GLFW_ICON", and if this is provided it will be loaded by GLFW and used as the windows icon.


[銷][臉書] 粉絲頁圖檔

Facebook will automatically crop your cover photo for mobile or desktop viewing, so it's important to make sure that you use the dimensions 820px wide by 360px for your cover photo size.

https://www.winwithteamwork.com/insights/facebook-cover-photo-size


Setting images causes error

First of all, thank you for maintaining this useful library.
The README says that in order to load some images:
pass.images.icon = iconFilename;
pass.icon(iconFilename);
await pass.images.loadFromDirectory('./images');

I understand that each of these lines can be used alone (and you don't have to call the first or second line followed by the third one), but when I use template.images.loadFromDirectory(directoryname) followed by pass.images.icon = filename, I get the following error:
error: Unhandled Rejection at: Promise {
   <rejected> { Error: EISDIR: illegal operation on a directory, read errno: -21, code: 'EISDIR', syscall: 'read' } } reason: Error: EISDIR: illegal operation on a directory, read

What am I doing wrong?

Also it is confusing that the pass file is created anyway. In my opinion, if an error occurs, the pass file should not be created and the error event should be triggered. Maybe this is not happening because you're not handling that promise's rejection.

There is a similar issue when the provided .pem file is not valid: an unhandled rejection exception is thrown. Could this be fixed as well?

@nickasd All these issues were fixed in 6.0 released today. Thank you!


Usage of [contrib.auth.models.]Permission instead of [helpdesk.models.]QueueMembership

This series of commits change the previous behaviour of "QueueMembership", which could be used to give access to certain ticket Queues, into a more ACL-like behaviour based on the Django contrib.auth package Permission.
To achieve this, a Permission is generated for each Queue, and the user is checked that permission in order to access a ticket.
In addition to the code, this PR includes an automatic migration for existing settings and also a management command to generate the permission in bulk. Once the appropriate settings flag is set, the new Queues will trigger the creation of their Permission.
Permissions can be assigned to either user or groups by the built-in User view in the admin (no changes needed). The Permission can be found under "helpdesk | Queue | Permission for queue: ", and internally has a codename of "queue_access_".
Removal of Queue remove the permission (given that the removal is not a batch removal).

The migration from perqueuemembership looks tight to me. I also think the usage of Django permissions is a little cleaner. I'll give this a little testrun, but so far
:thumbsup:

@alexbarcelo i can't seem to get the migrations to run. specifically, i am trying to migrate a db with some previous queue memberships (here is the file).
During ./manage.py migrate I get the following error
File "django-helpdesk/helpdesk/migrations/0009_migrate_queuemembership.py", line 29, in create_and_assign_permissions
    basename = q.prepare_permission_name
AttributeError: 'Queue' object has no attribute 'prepare_permission_name'

Could you have a look and make the migrations work?
Just by checking it out, I believe that using the RunPython function will give you the historical class, i.e. the old Queue that does not have the prepare_permissions_name function. You probably have to import the new class in addition, i.e.
from helpdesk.models import Queue as NewQueue
and then use NewQueue.prepare_permissions_name

Aaaaah! Sh**. Ok, I did this last commit, and didn't realize it may break the previous migration (as it actually did). I had tested the migration, but without the prepare_permission_name.
I think that it will be cleaner to add that prepare_permissions_name and the extras in adding required model extra info for permissions. I will check if reordering the branch is cleaner, maintining the general approach. Otherwise I will add this import NewQueue.

On a second note, I think you probably should provide a forward and a backward data migration.
And you should combine all of this into one single migration.
I.e.
class Migration(migrations.Migration):

    dependencies = [
        ('helpdesk', '0007_max_length_by_integer'),
    ]

    operations = [
        migrations.AddField(
            model_name='queue',
            name='permission_name',
            field=models.CharField(blank=True, help_text='Name used in the django.contrib.auth permission system', max_length=50, null=True, verbose_name='Django auth permission name'),
        ),
        migrations.RunPython(forward_data, backward_data),
        migrations.RemoveField(
            model_name='queuemembership',
            name='queues',
        ),
        migrations.RemoveField(
            model_name='queuemembership',
            name='user',
        ),
        migrations.DeleteModel(
            name='QueueMembership',
        ),
    ]

With this I was able to perform a migration and rollback and it seemed to work.
The migration backwards won't be able to fix group permission, because QueueMembership doesn't understand them, but typical scenarios will work ok, I expect.

Wonderful, thanks!
I'm going to merge this in so we can play with it a bit before doing a release with Django 1.9 / py3 / etc on Monday (ref #362)

@flinz Sorry, about joining the migrations, I haven't done it because I was fearing that it will break. I am not sure which order is done, and the migration requires some extra files to perform de RunPython and the deletion should not be done until the RunPython has finished.
I don't know what is advised in this situations. If performing a squashmigration is safe and you think it is the right way, I will add it --no problem about that. Simply that I was doubtful about correctness and stylishness

Thanks @rossp! I have played a bit more and I think that "it feels right", and migration should be straightforward. Maybe there are some bugs about ACL and malicious behaviour may break it. But the typical actions are ok, so I hope it only needs a bit testing to be on the safe side.
@flinz Sorry that I didn't elaborate on your comment about combining into a single migration. I was fearing that it would be strange because the RunPython requires the permission_name and should be executed before the QueueMembership objects aren't gone. I am not proficient enough on migrations, and this way seemed to be the "safe/conservative way" --although you already found me shooting myself in the foot :-/


Add detector params class

This class stores semi-static detector parameters such as rowreadtime and allows us to track it like a calibration file. It also creates a default calibration file by default on installation with the default values for these parameters.

I had to change the install behavior to create the configuration files for the pipeline on first import of corgidrp, which should happen any time we need to access a variable in the configuration file. So I don't anticipate it being an issue.


GPMDP does not work

I go into the program, and there is an endless download of the library, tried to reinstall, it  did not help (
Help me please.

Hey @PixelMass this is fixed in master, see the pinned issue #3660. There are pre-release builds available in the comments if you want to try those, otherwise you'll have to wait until the next release is out.


Current slayer task inconsistency

Please check if your issue is not a duplicate by searching existing issues
Describe the bug
After changing the current slayer assignment (outside of the Runelite client), the current slayer task GUI item at the top left of the screen stays as it was (I wouldn't think this part would be fixable). However, after killing enough of the new slayer enemy to get the combat bracelet notification, the number of remaining enemies in the GUI item is updated, but the type of enemy is not.
To Reproduce
Steps to reproduce the behavior:

Get a slayer assignment on the Runelite client, and note the current assignment GUI item at the top left (should be correct)
Outside of the Runelite client (on mobile, vanilla OSRS, OSB, etc), change the assignment (through skipping or completion)
Back in the Runelite client, note that the assignment GUI item has not changed
In the Runelite client, kill enough of the newly assigned enemy to get a combat bracelet notification
Note that the current assignment GUI item's image/text for the type of enemy has not changed, but the number has updated.

Expected behavior
After the combat bracelet gives its notification, the GUI item updates to show the new assignment's image and text.
Screenshots

Environment (please complete the following information):

OS: Windows 7
RuneLite version: <IP_ADDRESS>
Launcher version: 1.6.0

Additional context
I would suppose that you'd need to detect the enemy that the player just killed in order to trigger the combat bracelet notification. If that's not something you can do, this fix would probably be impossible.

we cant know what monster you're assigned unless you get the task on RL or check the task using gem/helm. so you gotta do that, theres no way around it

and although it's true that we could do some work to time the xp drop with the monsters that died to try to verify if our current idea of what task you are on is correct, that seems like a lot of work for a feature that would be pretty buggy and mostly unnecessary (if you either A just use runelite or B use your slayer gem when coming back to runelite from another client) - I think the wiki probably does a good job of explaining solutions A and B so I doubt there's much we can/should do a bout this

fixed in 1f6ae893d732cf3a022fe6d9662cd2a42ff2b217


Prepare release 2.4.0

Proposed changes:

Prepare release of 2.4.0 (with even more latest commits in main)

If another merge to main happens in between, I suggest creating the 2.4.x branch first and then releasing from there.

nice 🚀


Reddit post sentiment analysis

If a reddit post mentions a stock/coin, analyze sentiment

What Reddit posts do we analyze?
Options are:
r/wallstreetbets, r/stocks, r/StockMarket, r/investing, r/options, r/wallstreetbets
Should we only consider posts above a certain upvote threshold or consider all hot posts?

Maybe we can create a model like FinTwitBERT but specialized in Reddit posts

Otherwise, we could use a free sentiment model (BORING!)

Relevant datasets:

https://huggingface.co/datasets/winddude/reddit_finance_43_250k
https://figshare.com/articles/dataset/Wallstreetbets_Reddit_Data_10_2020_-_04_2022_/22010699
https://www.kaggle.com/datasets/aaaaaaaaade/reddit-wallstreetbets-hype-stock-posts?select=RedditAllBigClean+(2).csv
https://www.kaggle.com/datasets/thedevastator/unlocking-financial-opportunities-through-crypto
https://www.kaggle.com/datasets/leukipp/reddit-finance-data
https://data.mendeley.com/datasets/b6ns6d8xv3/1
https://www.kaggle.com/datasets/wordsforthewise/wallstreetbets-subreddit-data


Add back scipy as a dependency

Improve testing to actually make a GeoAxis

Checklist

[x] Used a personal fork of the feedstock to propose changes
[x] Bumped the build number (if the version is unchanged)
[ ] Reset the build number to 0 (if the version changed)
[x] Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
[x] Ensured the license file is being packaged.

closes #99

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.
I do have some suggestions for making it better though...
For recipe:

License is not an SPDX identifier (or a custom LicenseRef) nor an SPDX license expression.

Documentation on acceptable licenses can be found here.

@conda-forge-admin, please rerender

@dopplershift and @ocefpaf, are you good with this?

Thanks @xylar, makes sense since we want matplotlib we want to be able to use it :smile:

I'm going to take @dopplershift's comment https://github.com/conda-forge/cartopy-feedstock/issues/99#issuecomment-749106032 as tacit approval.


LANG-1400: Add StringUtils.mask() function

My alternative implementation for #332
See https://issues.apache.org/jira/browse/LANG-1400 for details

Coverage increased (+0.02%) to 95.247% when pulling c81ae1d9857fa84952ac763879beed4598d2b371 on stokito:LANG-1400 into 8e8b8e05e4eb9aa009444c2fea3552d28b57aa98 on apache:master.

Hello @stokito, as mentioned in #332, would you consider an extra parameter, to clearly specify the minimum number of masked characters?
From your implementation I can see that it is enforced, but it is limited, and it is not documented clearly either in the javadoc or by the parameter names.
The extra parameter can also make the method not being implicit on the unmasking of start/end (// we prefer to unmask from start and mask end if str len is smaller) and choose to equally subtract from unmaskedStart and unmaskedEnd to enforce the minimum amount of masked characters.

Hi @greenman18523

would you consider an extra parameter, to clearly specify the minimum number of masked characters?
For those use cases which I mentioned (masking credit cards and passwords) this looks not needed for me. Maybe you know some cases when this may be needed?

As I understood you are telling about more safety and do not unmask any symbol if incoming string is too short while implementation which I proposed will try to show at least some symbols from start.
For example mask("123456", 4, 4) = "12****" which makes hidden symbols more guessable.
But, to be honest, if someone uses so short password then it doesn't matter if it will be shown.
Another solution in this case we can mask everything when str len is 6 < unmaskendStart 4 + unmaskedEnd 4. I.e.  mask("123456", 4, 4) = "******". This is easier to understood but in the same time it still may be useful to unmask at least something but I don't think it's so critical.
What do you think about this proposition? E.g.
mask("12345678", 4, 4) = "********"
mask("123456789", 4, 4) = "****5****"
mask("1234567890", 4, 4) = "****56****"

I hope that unmaskedStart and unmaskedEnd in real life will be always reasonable (1-6) and the incoming string will be always bigger. We can actually restrict passing strings less that some length  and throw an exception.
But from possible use cases it looks that mask() function should be failsafe because it may be used just for logging of external input which can be anything and we shouldn't break it's processing. I even think about returning an empty string if null was passed.
Also we have to think about performance because I expect that the function will be widely used for in logging filters for any incoming request.

Hello @stokito

As I understood you are telling about more safety and do not unmask any symbol if incoming string is too short while implementation which I proposed will try to show at least some symbols from start.
For example mask("123456", 4, 4) = "12****" which makes hidden symbols more guessable.
But, to be honest, if someone uses so short password then it doesn't matter if it will be shown.

Yes, safety is my main concern. But in cases of arbitrary length data (e.g. names, addresses, messages), it's harder to say that one approach (the one implicitly specified in the method) on how many chars are masked or not is the correct. Also as a programmer I would like to have the flexibility to specify it, since regulations can differ world-wide.
But since your approach is meaningful in cases of specific-length data, I guess we can have two method, sharing one implementation.
public static String mask(final String str, int unmaskedStart, int unmaskedEnd, final char mask)
and
public static String mask(final String str, int unmaskedStart, int unmaskedEnd, int minMasked, final char mask)
with one calling the other.
I agree that the methods should be failsafe, otherwise we will need to place boilerplate code before calling them and I think it should be a one-liner, since as you say, main usage will be in logs.
Performance wise I think it is ok, and above all better to not rush on this matter, since it might not be needed.
P.S. I wouldn't use this for passwords, as anything can tip a malicious 3rd party, even the length. Better to not print anything.

Sound reasonable in the same time it will not always obviously which minMasked to use.
For example already mentioned credit card numbers in fact can have varying length as I remember up to 24 digits.
As I understood I created this function  for specific case which you had. Can you provide some examples from real life when minMasked was needed?

it will not always obviously which minMasked to use.

I am sorry, but I didn't understand this, can you please further elaborate? minMasked will be one, and it will be the absolute minimum of characters that will be masked.
Examples are SMS messages, e.g. some messages are short and contain one time passwords, or links to re-assign password. Such messages are generally very short, and since they have sensitive data, nothing can be logged, thus we have a minMasked value of 30. Also addresses, depending on how the user inputs it, this can be very few characters (usually just the street and the street number) or many (city, prefecture, country, etc), so when we print part of the address we need at least the first 15 characters to be masked.
The use cases might be specific, but considering this is part of a generic API the method should be flexible, in order to cover a wide variety of usecases.

Thank you for your examples. Just to clarify: maskedStart from this PR corresponds to minMasked from #332 but the discussion is about adding a  maxUnmasked, right?

some messages are short and contain one time passwords

it doesn't need for a masking at all. Short term generated values, OTPs and tokens like OAuth access_token (but not refresh_token) are safe to write to logs. If hacker stole logs we will have nothing to do with the data.

have a minMasked value of 30

Why exactly 30? What if sensitive data will be still shown in unmasked part? For this kind of data it's better to mask whole string.

when we print part of the address we need at least the first 15 characters to be masked

If I understood correctly, it's enough maskedStart of 15 and maxUnmasked not needed in the case.

a generic API the method should be flexible, in order to cover a wide variety of usecases.

What I'm afraid is that making the API too general leads to spending a time to read the docs, or incorrect understanding and misuse. As a developer I can clearly know how much chars to mask because I'm expect some kind of string but how much chars will be in real string is unknown variable for me, so maxUnmasked will be always a speculation. For example java's StringBuilder has a constructor with initial capacity and from my experience programmers are never even try to use it while this is critical for performance. So adding a new parameter maxUnmasked may be not so useful in real life but confusing.

Just to clarify: maskedStart from this PR corresponds to minMasked from #332 but the discussion is about adding a maxUnmasked, right?

I don't see a maskedStart parameters, these are the parameters I see in this PR: final String str, int unmaskedStart, int unmaskedEnd, final char mask
If you are referring to unmaskedStart, then this is a parameter with a dual, non-obvious role, which is not good. There should be different parameters for different roles. If I a missing something from this PR, please elaborate.

some messages are short and contain one time passwords

it doesn't need for a masking at all. Short term generated values, OTPs and tokens like OAuth access_token (but not refresh_token) are safe to write to logs. If hacker stole logs we will have nothing to do with the data.

No, you must not write any sensitive data, even if they are valid for half a nanosecond. Every tiny info a 3rd party gains gives them more leverage on how to further proceed. Hackers don't always steal old logs, they can live monitor your logs also! For OTPs, there should be different endpoints where logging of messages is disabled. But, just in case something like that goes through the main channels. 30 is a best-effort estimation, based on the currently seen patterns for OTPs.

a new parameter maxUnmasked may be not so useful in real life but confusing.

I am referring to a minMasked parameter, which by it's name states it's purpose. Which is the absolute minimum number of characters that must be masked.

Sounds reasonable in the same time it will not always obviously which minMasked to use.
For example already mentioned credit card numbers in fact can have varying length as I remember up to 24 digits.
As I understood I created this function for specific case which you had. Can you provide some examples from real life when minMasked was needed?

FYI, card numbers can vary from 13 to 19 digits. PCI DSS allows to log PANs mask in such way:
Here is proper maskifying Scala code:
    val START_SYMBOLS = 6
    val END_SYMBOLS = 4
    val TOTAL_SYMBOLS: Int = START_SYMBOLS + END_SYMBOLS

    private def stars(count: Int): String = "*" * count

    def maskify(number): String =
      if (number.length <= 6) {
        number
      } else {
        val maskLength = (number.length - TOTAL_SYMBOLS).max(0)
        val numberFromEnd = maskLength + START_SYMBOLS
        number.substring(0, 6) + stars(maskLength) + number.substring(numberFromEnd)
      }

As a result:
4242424242424242     ~> 424242******4242
424242424242424242 ~>424242********4242

Any plans on this?

Needs discussion, see mailing list WRT to what belongs in Commons Lang vs. Commons Text.

Closing: No action in 4 years. See reference to Commons Text. Also Java has stock support for replace methods and regular expressions.


JAMES-2586 Implement Deleted Message Vault backed by Postgres

resolve https://github.com/linagora/james-project/issues/5000
TODO: plug DeletedMessageVaultDeletionCallback.

I am missing a test...

I thought about it as integration test DeletedMessageVaultIntegrationTest as part of the https://github.com/linagora/james-project/issues/5005.
But I can do the the vault IT here if you want ^^

Maybe we can modify SinglePostgresConnectionFactory to be backed by the pool connection (and rename it)

That would be enough to me. No need of an extra integration test then.

read it

Rebase needed...

The file /home/tungtv/workplace/JAMES/1james-project/server/protocols/webadmin-integration-test/postgres-webadmin-integration-test/pom.xml is not sorted -> [Help 1]

@Arsnael Can we merge this?


Log::useDailyFiles() duplicates log entries.

Issue is when Laravel Log:: is used in a loop while using custom daily files:
    public static function push($context = "general", $message = "", $level = "info", array $data)
    {
        Log::useDailyFiles(storage_path().'/logs/API_'. strtoupper($context) .'.log', 7);
        Log::log($level, $message . "\n" . json_encode($data));
    }

            foreach($log as $entry){
                CustomLog::push($entry['sender'], $entry['action'], 'info', ['data' => $entry['data']]);
            }

Logs are saved multiple times in defined log file, for example if loop is going thru 3 elements contents of a log is:
Log element 1
Log element 2
Log element 2
Log element 3
Log element 3
Log element 3

Is this issue with Monolog or is it on Laravel side?

Isn't that just because Log::useDailyFiles() pushes a new log handler to Monolog? So the first time you run it you add a handler, and ask it to log your data. The second time, you push a handler, and ask your old handler to log your data, then ask the new handler to log your data.
You should just register your log handler once during bootstrapping, and push your data.


Implementation Plan: Use a single index for all API results

Description

After consulting with some Elasticsearch experts, we've determined that we may be able to improve Elasticsearch performance by using a single index with a sensitive keyword rather than two indices (filtered & unfiltered). The existence of two indices that are being regularly queried means that Elasticsearch must be loading records off disk from both indices much more frequently than it would do so on its own. The difference in disk ops between querying two indices and only one index can be seen in this graph:

The easiest way to accomplish this might be an extra step during the initial indexing when building the document to feed into Elasticsearch which uses the sensitive terms list to determine a sensitive keyword field on the document.
It might also be possible to build and index the documents in a first step and default to sensitive=False, then process all the documents in a second pass once the index is built. This could query the documents matching the sensitive terms list and update the sensitive field for those documents (since we've seen they're about 1% of the total index size). We could use the bulk update API for this, although we've been discouraged from doing so since that would be a very long-running request against the cluster.
Alternatives

We can leave the cluster as-is with the two index setup; it has not been shown to affect the periodic response time spike we've been seeing recently.
Additional context

The original outline for the two index approach can be found here: https://docs.openverse.org/projects/proposals/trust_and_safety/detecting_sensitive_textual_content/20230330-implementation_plan_filtering_and_designating_results_with_sensitive_textual_content.html

@AetherUnbound I was wondering if you had any thoughts about when we might prioritise this? I think it would have benefits for service reliability, particularly with index redundancy, if we had fewer indexes spread across the data nodes (we could increase the number of replicas, for example, so that no single node carries a significant amount of any one index).
I also wanted to record some thoughts I had today about how this could work. In particular, regarding the new sensitive field on the index. I'm not sure if you used "keyword" intentionally, Madison, but it would be great to have it actually be a keyword list field, mirroring the "sensitivity_reasons" that we return from the API. So if the work is sensitive due to a confirmed use report, we would add user_reported to the sensitive keyword list. If the work has sensitive text, detected during ingestion, we would add sensitive_text to the sensitive keyword list on the work.
When filtering for non-sensitive works (the default behaviour), instead of using mature=false in the query, we'd query for works where sensitive is an empty list. This makes the sensitivity reasons on the API response trivial construct. It would save at least one ES query for every search, and could save database queries as well for image results and audio results without waveform data.

That's a good question - I imagine it would affect the work being done as part of #383. Do you think it would make sense to include this in that effort at all, or should it be a separate prioritization?

Can you specify how you see the relation to the other project? The moderation workflows project wouldn't effect the way user reported sensitivity is detected during ingestion (which is done by checking for an entry in the SensitiveMedia tables). That's remaining the same in the moderation workflows project.
I think this one could be done agnostic of the moderation workflow changes, but I might be missing a connection between them.

I see! I was thinking it might be involved because both (seemingly, to me) involve modifying the mature/sensitive aspects of the Elasticsearch index, so it might be more ideal to have them occur at the same time. If they can be done distinctly, then perhaps that's best.

Gotcha. #383 doesn't actually won't change anything about the way we handle mature/sensitive in our indexes, we'll keep using the same method of reading from the sensitive media tables at indexing time. I think this IP would augment that process with additional checks on the sensitive terms list, in addition to potential changes to how we represent that information in the index (as I suggested above).
#383 doesn't touch the ingestion server at all in fact, and doesn't modify search either, so this IP and that project can happen in any order without interrupting each other.

@stacimc Heads up that it would be good to keep this IP in mind during the ingestion server removal project. Mainly just want to make sure that we don't make it harder to remove the double-index approach than it is now. I'd be surprised if we managed to make it harder, but worth checking to be sure.

Just dropping a note on how to potentially reduce the subset of works that need to be scanned, it's theoretically possible for us to exclude a subset of providers from sensitive content detection, I think. Nappy and WP Photo Directory for example are definitely safe to skip. Their datasets are relatively small, but even then, I think from a perspective of trying to reduce the false positives, it's a good idea to not use our naive text analysis method when we know that the provider is safe.
We could implement this in the existing reindex query by adding an OR clause to the boolean when creating the filtered index: source in <trusted subset> OR <existing terms query>
For a single index approach, it would just mean skipping whatever text analysis we end up doing when the source is in the trusted subset.

Just wanted to share some notes I'm working from to start off with, mostly to help clarify the requirements of this change and some potential areas I will evaluate for implementation.
Requirements:

Arbitrary changes to the list of sensitive terms continue to be reflected in search within one data refresh cycle

Keeping it "within one data refresh cycle" means the responsiveness to changes does not go down, but doesn't preclude it somehow becoming faster than one data refresh cycle, if for some reason that's possible

A single index rather than a filtered and unfiltered index
No ambiguity between the existing mature field and however we indicate textual sensitivity in the index
No negative change in search performance (i.e., cannot make search worse off after this)
Including or excluding results with sensitive terms must not affect ranking

Not requirements:

That textual sensitivity exist in the catalogue, as opposed to only in Elasticsearch (it is not currently the case and making that a requirement would, strictly speaking, expand the scope of the project)
Excluding a subset of "safe" providers from sensitive text scanning

Not being a requirement does not exclude the possibility of the thing, if it turns out that works in our favour. I just want to make sure the scope of requirements is clear.
Potential methods:

In the catalogue, initially for all new incoming works, and then as an ongoing DAG that does a batched update on all works. The DAG is ongoing because it is reused whenever we made changes to the sensitive text list, which the DAG itself checks for (e.g., the DAG pulls the list, if it matches the list the last time the DAG ran (maybe based on some hash), then skip, otherwise, run the batched update)

Could produce an inconsistency if this batched update overlaps with a data refresh and the data refresh inserts from the catalogue into the API DB when only half of the works have been re-evaluated; this would technically reduce the responsiveness for the whole data set to two data refreshes, rather than one data refresh.
This seems like the least likely to fit into the requirements to me.

In the data refresh filter data step and include sensitive text designation in the API DB, to then be inserted upstream to Elasticsearch as a 1-to-1 mapping.

Potentially makes it easier for moderators to know whether the work has sensitive text, compared to having it only in ES? Though querying it from ES is trivial so not a hard benefit.

In the indexer worker when creating the document for Elasticsearch to index.

Could potentially benefit from higher available performance throughput of the indexer workers compared to the local executor of the Airflow box, where a lot is already happening?

In some other process that produces an Elasticsearch enrichment data set that gets merged into works during indexation.

I don't know how this would handle new works, or whether there's any conceivable benefit to this, or whether it even actually fits into the use case intended of Elasticsearch's enrichment step.

I appreciate you enumerating a few different approaches here - I think I'd also be biased towards the indexer worker approach, given we aren't trying to dismantle those and shunting that work on separate machines rather than the Airflow schedule (as you point out) will probably make things easier to manage!

You know, that was the assumption I was working from too (that the indexer worker is sticking around for the foreseeable future), but then I learned about Elastic connectors and this tool Pgsync (they are separate things) and it made me wonder whether the indexer worker approach somewhat "reinvents the wheel" when it comes to loading data from PG into ES.
I'm not going to weigh that into this decision, I think, because I've only just learned about it and I don't actually know whether it really is a suitable replacement for the indexer worker or whether that approach is something we could conceivably switch to even in the medium-term... but it's worth mentioning to challenge that assumption (it is just an assumption, nothing fundamentally necessary about it, apparently) and use it as a framing device to evaluate the requirements and implementation to ensure it is described flexibly enough to support transformation at a different stage. In other words, that the intention and method are abstracted from whatever location in our code and current processes we decide on. I don't know how that would manifest at this stage, obviously, but thought I would share it. Only just learned about those two tools (Es connectors, Pgsync) in the last few days.

Just wanted to share some notes from my research and experimentation today. There is no action item in this comment, I am just sharing what I have learned and a rough outline of a plan to test something in the staging cluster.
Today I revisited Elasticsearch's documentation on ingest pipelines. I actually originally started by reading about analysers, because I wondered if there was some way for an analyser to designate a token with additional attributes based on some condition. I wondered if it would be possible to designate the sensitive terms as such at the token level. I wasn't able to find anything about that, but it eventually led me back to the ingest pipelines documentation. Apparently a dedicated ingest node is only necessary if you have "heavy" pipelines. I'm not totally sure what qualifies as a "heavy" pipeline, so I figured it would be worth trying out in staging, just in case this ends up being workable.
I wanted to better understand how pipelines work, so I created the following pipeline in my local development cluster:
{
  "description": "Build the sensitivity designation array",
  "processors": [
    {
      "append": {
        "description": "Textual sensitivity",
        "field": "sensitivity",
        "value": [
          "sensitive_text"
        ],
        "if": "def re = /water|bird|running/; if (Objects.nonNull(ctx.title) && ctx.title =~ re) { return true; } if (Objects.nonNull(ctx.description) && ctx.description =~ re) { return true; } if (Objects.nonNull(ctx.tags) && ctx.tags.stream().anyMatch((t) -> t.name =~ re)) { return true; }"
      }
    },
    {
      "append": {
        "description": "User reported sensitivity",
        "field": "sensitivity",
        "value": [
          "user_reported_sensitivity"
        ],
        "if": "ctx?.mature"
      }
    }
  ]
}

The Painless script (what goes in the if of an ingest processor object) has to be a single line in this request document, so here it is expanded out like you would normally see:
def re = /water|bird|running/;

if (Objects.nonNull(ctx.title) && ctx.title =~ re) {
  return true;
}

if (Objects.nonNull(ctx.description) && ctx.description =~ re) {
  return true;
}

if (
  Objects.nonNull(ctx.tags)
  && ctx.tags.stream().anyMatch((t) -> t.name =~ re)
) {
  return true;
}

Painless is very close to regular Java, so was pretty easy to work with. Because I have a (long ago) history using Java professionally, I was familiar with the way the documentation worked and most of the concepts, so writing the script was easy. While the stream API is nice and pretty readable, using a stream probably isn't necessary and a for loop might be better for performance when this runs over hundreds of millions of documents.
The append processor appends a value to the list at the specified field. If the field/list does not exist, it will create it. So the pipeline above will create and append the two in-use sensitivity designations that the API currently emulates at request time.
To query for non-sensitive items, we would use the exists query, and negate it. In query string syntax, that ends up being !_exists_:sensitivity.
To make the pipeline work, I needed to add it to the index settings. To make the field available in queries, I had to add it to the index properties.
diff --git a/ingestion_server/ingestion_server/es_mapping.py b/ingestion_server/ingestion_server/es_mapping.py
index d7ec3272f..ebbf68506 100644
--- a/ingestion_server/ingestion_server/es_mapping.py
+++ b/ingestion_server/ingestion_server/es_mapping.py
@@ -19,6 +19,7 @@ def index_settings(media_type: MediaType):
             "number_of_shards": number_of_shards[media_type],
             "number_of_replicas": 0,
             "refresh_interval": "-1",
+            "final_pipeline": "sensitivity",
         },
         "analysis": {
             "filter": {
@@ -73,6 +74,7 @@ def index_settings(media_type: MediaType):
             "source": {"type": "keyword"},
             "filetype": {"type": "keyword"},
             "category": {"type": "keyword"},
+            "sensitivity": {"type": "keyword"},
             # Text-based fields
             "title": {
                 "type": "text",

I then deleted my local indexes and just re-ran ov j api/init and everything worked!
Because of how simple this is to get a basic version of, I'd like to test it out in the staging index. To do so, I will follow these steps:

Create the ingest pipeline in the staging cluster, but use the actual sensitive terms list. I will create the Painless script for the sensitive text processor using a template to interpolate the real sensitive terms list into the regex on the first line. The real test for this is whether such a huge regex will perform acceptably. Considering the approach would be theoretically identical to what we would do in the indexer worker, I would be surprised if that was the bottleneck.
I will trigger the creation of a new image index using the create_new_staging_es_index, with the following index_config: { "settings": { "index": { "final_pipeline": "sensitivity" } }, "mappings": { "properties": { "sensitivity": { "type": "keyword" } } } }. That should be enough to make sure the new index uses the ingestion pipeline.
Then I will monitor the resource usage of the staging index to ensure it isn't thrashing. I'll compare the usage to a regular data refresh's create filtered index step to see how it compares to the only other reindex query we currently use.

If that goes well, then I would want to use the recreate_full_staging_index DAG after pushing the added index configuration to the ingestion server/indexer worker (whichever is relevant at the time) conditionally applied only for the staging target environment. Running the test with a full staging index creation will be more representative of the real approach. It might be worth going straight to this, but because it requires committing code, I want to avoid it, at least to see whether the hugeness of the regex statement will be an issue for the live Elasticsearch cluster.
I am out of time today, but I will also of course test the large regex statement in my local dev cluster first, in case there is some kind of size constraint and other workarounds would be required for it. I will also swap the stream API out for a regular for loop.
As for the possibility of using an ingest pipeline and managing the code/configuration for it, we can just use Airflow for it. Provided there is not an issue with the size of the regex, this might be a feasible approach and will probably outperform the indexer worker version. I believe that based on Elasticsearch's claims that Painless scripts are highly optimised for their purpose, and because they are widely used, from what I can see online. Additionally, understanding Elasticsearch ingest pipelines may be of great use for future search relevancy work.
Anyway, just sharing what I learned today. I will share a concrete plan and timeline of my experimentation in staging with the core maintainers in Slack, with the goal of trying it out next week if possible.

Okay, I've got a version of the ingest pipeline that works with the full sensitive text body of terms.
I ran into some issues when testing it locally. I first tried a regex that was just "|".join(terms) (with some additional escaping and such), but ran into ES's regex limit-factor, which you can read about here: https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html#regex-circuit-breaker. TL;DR: there is a limit to the number of characters a regex is allowed to consider, and with a long regex and long string, ES's default settings aren't happy. We can change that number, but it's probably best not to.
So instead of a regex, I landed on a version that just iterates through the strings. To get around problems with unicode escaping in a string passed to a Python template string, then passed to ES in a JSON blob (lots of layers), and to make the script a little less jarring to look at when rendered, I base64 encode the sensitive terms list.

This is the long JSON blob. Do not decode the base64 string unless you are prepared to look at the real sensitive terms list
{
    "description": "Build the sensitivity designation array",
    "processors": [
        {
            "append": {
                "description": "Textual sensitivity",
                "field": "sensitivity",
                "value": [
                    "sensitive_text"
                ],
                "if": "\nString termsb64 = \"[encoded data, 24208 characters]\";\nString[] terms = /\\n/.split(termsb64.decodeBase64());\n\nif (Objects.nonNull(ctx.title)) {\n    for (term in terms) {\n        if (ctx.title.contains(term)) {\n            return true\n        }\n    }\n}\n\nif (Objects.nonNull(ctx.description)) {\n    for (term in terms) {\n        if (ctx.description.contains(term)) {\n            return true\n        }\n    }\n}\n\nif (Objects.nonNull(ctx.tags)) {\n    def tags = new ArrayList();\n    for (tag in ctx.tags) {\n        tags.add(tag.name);\n    }\n\n    for (term in terms) {\n        if (tags.contains(term)) {\n            return true\n        }\n    }\n}\n"
            }
        },
        {
            "append": {
                "description": "User reported sensitivity",
                "field": "sensitivity",
                "value": [
                    "user_reported_sensitivity"
                ],
                "if": "ctx?.mature"
            }
        }
    ]
}

This works locally, so I'm going to push the pipeline to the staging cluster, and then start the staging es index creation DAG with the additional settings to add the sensitivity field in the properties and the pipeline to the index settings.

Okay, I've tried to run it. It turns out I misunderstood the Elasticsearch documentation. It isn't required to have a dedicated ingest node, but you do have to have nodes configured with the ingest role to use ingest processors at all.
Maybe something for us to explore in the future, but not now. We need to keep our cluster as simple as possible to maintain, and adding additional nodes or mixing data and ingest nodes is just more complexity than is wise to use now.
I'll write the IP recommending the functionality be added to the indexer worker.


Use correct colorize method

Fixes #223.
Details
We merged #183 recently, but I missed the method on line 156... 🙇♂
Move dependency to bullet_train-base gemspec?
In Issue #222, developers who don't already have colorize on their machine will get a cannot load colorize error. I mentioned putting colorize in a gemspec in PR #183, specifically for bullet_train-base. Since we run bundle install later in bin/configure, it would make sense to have it as a run-time dependency that's already downloaded before the developer runs bin/configure. So we can either...

Remove colorize from Gemfile here and move it to the bullet_train-base gemspec
Revert to the manual methods we had in the first place.

If option 1 seems reasonable, I'll make a PR in bullet_train-base.
Tagging @esmale for reporting.

Fixed this before noticing this PR!

@andrewculver Np 🙌


Updated YAML format to support different enumerated values for R/W

The previous YAML format didn't support separate enumerated values for read and write usage.  This is required e.g. here http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0553a/Cihcajhj.html
I updated the peripheral yaml files that used the old format and the README example.  There was also quite a funny typo in the README :hankey: :scream: :see_no_evil:

Thanks, lgtm!


Adjust state dropdown copy to direct to newsletter signups

Overview
We're getting a lot of emails from people who don't see their state in the dropdown (which is helpful for prioritizing). Typically, we direct them to sign up for the DB newsletter in order to get an update when their state is added. This PR places that signup earlier in this process so that people know what to expect.
Checklist

[x] Description of PR is in an appropriate section of CHANGELOG.md and grouped with similar changes, if possible

Demo

Notes

This is what the Mailchimp form looks like now:

Testing Instructions

git pull
Create a new map
Note the new helper copy for the "State" field. Click the link. It will open the newsletter signup in another tab, which now has a new text input.

Closes #549

LGTM!

LGTM!

After some discussion with @jfrankl, we edited the copy in the Mailchimp form to make this question more general:

After some discussion with @jfrankl, we edited the copy in the Mailchimp form to make this question more general:


Update README.md

Current coverage is 93.72%

Merging #55 into master will decrease coverage by -0.20% as of 30cff11

@@            master     #55   diff @@
======================================
  Files            6       6
  Stmts         2173    2120    -53
  Branches         0       0
  Methods          0       0
======================================
- Hit           2041    1987    -54
  Partial          0       0
- Missed         132     133     +1

Review entire Coverage Diff as of 30cff11

Powered by Codecov. Updated on successful CI builds.

Hmm ok - Now we shouldn'tshouldn't forget to update the Readme


created my html file

Your pull request looks good, and is ready for merging!
Thank you for being a part of Open Source, I hope you continue to help other repositories in the future!
This is an automated response, beep boop.
if you think there is a mistake, contact a maintainer

fbd8238bda027087bc66c99b342e4d8ba0828cdb


Rmsprop optimizer

DeepMind is using RMS prop in Video Pixel Networks  as nakosung pointed out over there. Video Pixel Networks are related to Wavenet.
I figured I would try RMS prop out and see if a miracle ensues.

Does it help?
It would be nice to have a more generic optimizer selector rather than all these hardcoded constants. How about a dictionary that maps names to functions, and the argument parser is given its .keys()? I don't know what to do with the momentum though. Maybe we need a separate function or class that handles this. Or, at least, WaveNet could export a list with the supported optimizers rather than read the individual globals from the outside.

This is on top of #107, right?

Yes, on top of 107.

more generic optimizer selector rather than all these hardcoded constants. How about a dictionary

Yes, that's a better idea.

Does it help?

I haven't started running it yet. I thought I could just switch the optimizer and resume training. But it fails to load the checkpoint because there are tensors specific to the optimizer it tries to load and can't find. I think we must be saving more than just the trainable vars in our checkpoints. Is that intended, or can we change that?

Here's comparison training on speaker p280. Orange line is SGD (learn rate = 0.01, momentum = 0.9), aqua is adam (learn rate = 0.001), purple is RMS Prop (learn rate = 0.001, momentum=0.9).

There's something wrong with the diff, it looks like your branch undoes PR's that have already been applied and then it re-applies them. If will not have any effect because of the squash, but it makes it hard to see the changes. Could you try to fix it? I believe this is GitHub's fault, for being too smart with the squash merges. You may have to undo your internal merge, rebase the old branch to the current master, and merge again.

@jyegerlehner, have Deep mind or others open the Video Pixel Networks' source codes? Recently I am reading this paper. Thanks.

@ccienfall Not as far as I know.


Liltoon Unlit Bug

bug in all version
I tried to reinstall unity and does not work.
I was using this for avatar brightness
the lightning works in unity but It does not work in vrchat
the animation breaks.

I tried to click fix now but it goes back to zero.

it only works in unity

here are my settings

Fix Now button fixes the property value, not animations.
First, you need to isolate the cause of the problem. Add BlendShapes to that animation and see if the animation plays in VRChat. If it doesn't play, it's most likely a problem with the Animator, ExpressionParameters, or ExpressionsMenu.


chore(deps): update execa to v9

Closes #1846

Maybe this change broke the Cloudflare Pages action? 😥

I've fixed it on cloudflare pages.
That was node.js version issue of cloudflare pages.


Objstore: Bump Objstore for Azure Workload Identity support

[x] I added CHANGELOG entry for this change.
[x] Change is not relevant to the end user.

Changes
Bump objstore, adding support for authentication using Azure Workload Identity
Verification
Testing performed as part of https://github.com/thanos-io/objstore/pull/82

@yeya24 If you're happy with this, could it be reviewed/merged? Tagging you as you merged the PR in objstore.

Can you help update the doc of https://thanos.io/tip/thanos/storage.md/#azure? @rikhil-s I think Azure client doc is outdated. msi_resource is deprecated and we need to mention that workload identity is used when user_assigned_id is not set (IIUC)

Ok, I will update the docs.

Can you help update the doc of https://thanos.io/tip/thanos/storage.md/#azure? @rikhil-s I think Azure client doc is outdated. msi_resource is deprecated and we need to mention that workload identity is used when user_assigned_id is not set (IIUC)

@yeya24 Have now updated the docs.

Umm, I saw the changelog was moved to v0.33 release. Is it intended?
Let's keep it in master.

Umm, I saw the changelog was moved to v0.33 release. Is it intended? Let's keep it in master.

Can this change be in the 0.33 release?

Umm, I saw the changelog was moved to v0.33 release. Is it intended? Let's keep it in master.

Can this change be in the 0.33 release?

yes ill cherry pick the commit into the 0.33 branch later

Umm, I saw the changelog was moved to v0.33 release. Is it intended? Let's keep it in master.

Can this change be in the 0.33 release?

yes ill cherry pick the commit into the 0.33 branch later

Ok thanks, I'll move the changelog entry back to the Unreleased section in this MR then.