[WIP] sokol_spritebatch.h

A work-in-progress XNA-style spritebatch library on top of sokol_gfx.
Sprites are batched based on their texture ID. The library iterates over all the sprites submitted and creates a batch when the texture of the current sprite is different from the previous sprite. Making sure the same texture is used as often as possible, via texture atlas and/or via appropriate use of sort modes, is good for performance.
 There is also the option of doing no sorting at all (SB_SORT_MODE_DEFERRED), which is useful if you know that the sort would be redundant, like if you are using the library to render a single fullscreen quad. Sorting by depth can be used to achieve nice visual effects like Y-based sorting in a top-down game or background layers in a sidescroller.
Shaders and pipeline are currently taken care of internally, but eventually an API will be provided to allow users to override the default pipeline with a custom one. This will enable users to change things like the blend state or shader being used, which will be advantageous for developing custom effects like additive sprite-based lighting or applying post-processing to a fullscreen quad.
Like XNA, this library relies on premultiplied alpha for alpha blending. Ideally this would be handled offline by your project's content pipeline, however, a function is provided (sb_premultiply_alpha) to post-process texture data before it is used to create an sg_image.

Hey @nyalloc,
Coming from XNA/MonoGame, I would advise against having a batcher which sorts draw calls. I would consider sorting a separate concern to merging/batching draw calls for textured quads.
Christer Ericson has blogged about sorting draw calls:  https://realtimecollisiondetection.net/blog/?p=86. As you can see sorting draw calls is something that will be more or less unique to the needs of the game. I.e., it is likely to be more than just sorting by texture/depth. XNA's SpriteBatch is a good beginner friendly introduction API for getting 2D to "just work". However, developers often quickly get frustrated that the options for sorting don't really meet their needs in the end and thus will only use "Deferred" mode and fallback to do their own sorting. Not having a sort feature and focusing solely on merging draw calls of textured quads simplifies the code and the API significantly.
Additionally, some ideas floating around for some improvements for a better API for XNA's SpriteBatch:

Use a 3x2 matrix for a transform simplifying the data needed for a "batch item". This removes position, rotation, origin, scale, etc. This also solves the problem of easily having parent relationships of sprites since it's just a problem of constructing the correct matrix. It also simplifies the code for transforming the 4 vertices of the quad significantly; e.g. less "if" checks.
Caching the internal vertex buffer for re-use in later frames. When drawing more or less static geometry like maps in 2D games, the geometry does not change frame-per-frame, or at least it is unlikely to change frame-per-frame. It is a waste to fill the vertex buffer every frame with data that effectively does not change frequently; waste for the CPU to crunch through the data to enqueue the sprites and also a better hint could be directed to the GPU of using DYNAMIC vs STREAM.

Just my 2 cents.

That looks great! The for your work on this.
A user-defined sorting key would be great (for shaders and other states). Another example of a unique spritebatcher is
https://github.com/RandyGaul/cute_headers/blob/master/cute_spritebatch.h
It builds the sprite atlas as well.

Hello! Back on this now. As per @lithiumtoast feedback I've removed the sorting API, right now users will be capable of worting their sprites externally in a manner that is appropriate for their application. I can always reintroduce this if there is ever demand for it, but removing it does simplify the API and the implementation quite a bit.
New in these changes are the ability to render to different render targets + use different sg_pipelines. I'm still working on these parts of the API and I think it will be a little while before I get them right. Luckily there is plenty of examples of this in the other sokol utility headers.
Regarding the 3x2 matrices, I think this is an interesting idea. @NoelFB uses a 3x2 matrix stack for his spritebatcher and this is something I can see being quite handy without adding very much overhead (3x2 matrix multiply against 2d points is pretty trivial).
I'll continue working on this and throw up a sample application when it is ready.

Btw, trying to do this myself with sokol-shdc. GLSL uses column major so it needs to be a mat2x3 not a mat3x2. 3x2 is for row major.
Row major layout sequential (XNA/MonoGame/FNA/DirectX/.NET):
m11 m12 m13 m21 m22 m23

Column major layout sequential (OpenGL):
m11 m21 m12 m22 m13 m23

I'm working with bindings for C# so my matrices are laid out in memory with row major, hence why I said 3x2.
Here's a image visually explaining the affine transforms possible for 2D using row-major. Notice the greyed out row which is always the same. This is why a 3x2 matrix is more compact for 2D instead of just a 3x3.

It doesn't really matter if sokol's shader compiler struggles with 3x2 matrices, I won't need them for the shaders. If I introduce a 3x2 matrix stack to this, then that multiplication will be done CPU-side when generating the vertex buffer. This is fine, because the 3x2 matrix multiplication is really trivial. NoelFB's batcher demonstrates this pretty well.

Ah, I was going for doing it all in the vertex shader:
...
layout(location = 0) in mat2x3 vs_Matrix3x2; // SG_VERTEXFORMAT_FLOAT3
// location = 1 is used by the 3x2 matrix // SG_VERTEXFORMAT_FLOAT3
layout(location = 2) in vec4 vs_SourceRectangle; // SG_VERTEXFORMAT_USHORT4N
...
const vec2 corners[4] =
{
    vec2(0, 0),
    vec2(0, 1),
    vec2(1, 1),
    vec2(1, 0)
};
...
mat4x4 modelMatrix = mat4x4(
        vec4(vs_Matrix3x2[0][0], vs_Matrix3x2[0][1], 0, 0),
        vec4(vs_Matrix3x2[0][2], vs_Matrix3x2[1][0], 0, 0),
        vec4(0, 0, 1, 0),
        vec4(vs_Matrix3x2[1][1], vs_Matrix3x2[1][2], 0, 1)
    );
...
gl_Position = ViewProjectionMatrix * modelMatrix * corners[gl_VertexIndex] * vs_SourceRectangle.zw * 65535;
...

That is neat. The vertex buffer takes the form of an array of matrices that define the sprite dimensions / transform? I wonder how you would handle texture coordinates or other per-vertex state.
Right now it's not clear to me if there would be a meaningful performance benefit to this appraoch so I'm not going to change course. A worthwhile experiment though. Let me know how it goes.

The vertex buffer takes the form of an array of matrices that define the sprite dimensions / transform?

By using instancing, the single vertex buffer is the "sprite batch items"; each vertex is the instance data for the textured quad.

I wonder how you would handle texture coordinates

It's done by using the source rectangle position and size along with a uniform for the texture size. However, the consequence is that sprite batching would have to be fragmented by texture. Thus, using a different texture requires stopping the current batch in process and starting a new batch. For me, this is acceptable because I would be using a texture atlas.
Here's the whole shader program I'm using with sokol-shdc. Note that I'm using bottom-left as origin for model/world positions. What's really nice is that by moving everything into the vertex shader, the code for the CPU is dead simple as just filling data into the "sprite batch items" array and then uploading it to the GPU via the vertex buffer.
@vs vs

layout(binding=0) uniform MatrixData
{
    uniform mat4x4 ViewProjectionMatrix; // [View To Projection]x[World To View]
};

layout(binding=1) uniform TextureData
{
    uniform vec2 TextureSize;
};

layout(location = 0) in mat2x3 vs_matrix3x2; // GLSL uses column-major meaning `a`x`b` is `a` columns by `b` rows. However, people usually think of a matrix as `a` rows by `b` columns  when they say `a`x`b`.
// location = 1 is used by the 3x2 matrix because in sokol we can only use vectors to which the matrix is 2 x vec3.
layout(location = 2) in vec4 vs_source_rectangle;
layout(location = 3) in vec4 vs_color;
layout(location = 4) in float vs_depth;

out vec4 fs_color;
out vec2 fs_uv;

const vec2 corners[4] =
{
    /* Rectangle from two clockwise triangles:
        0, 1, 2) and (0, 2, 3)
        NOTE: Using [-1, 1] so the origin (0, 0) is the centre of the rectangle; this makes it so rotation by default is around the centre of the rectangle
	    1---------2
	    |       / |
	    │     /   |
	    │   /     │
	    │ /       │
	    0---------3
	 */
    vec2(-1, -1), // bottom-left
    vec2(-1, +1), // top-left
    vec2(+1, +1), // top-right
    vec2(+1, -1)  // bottom-right
};

void main()
{
    /*  the 3x2 matrix has the data layout: m11, m12, m21, m22, m31, m32
        however, they are copied from CPU to GPU like so into vectors
        ----0---1---2-
        0| m11 m12 m21
        1| m22 m31 m32
        we actually want to use the 3x2 matrix as a 3x3 matrix like so (column-major order)
        ----0---1---2-
        0| m11 m12  0
        1| m21 m22  0
        3| m31 m32  1
        so that we can build the 4x4 matrix from that 3x3 matrix like so (column-major order)
        ----0---1---2---3-
        0| m11 m12  0   0
        1| m21 m22  0   0
        3|  0   0   1   0
        4| m31 m32  0   1
        thus we have the following matrix in the code below for mapping between the blittled data
        with GLSL we index matrices by column then row starting at 0 not 1, e.g. m12 = m[1][0]
        ----------------------------------
        | d[0][0] d[0][1] 0______ 0______
        | d[0][2] d[1][0] 0______ 0______
        | 0______ 0______ 1______ 0______
        | d[1][1] d[1][2] 0______ 1______
    */

    mat4x4 model_matrix = mat4x4(
        vec4(vs_matrix3x2[0][0], vs_matrix3x2[0][1], 0, 0),
        vec4(vs_matrix3x2[0][2], vs_matrix3x2[1][0], 0, 0),
        vec4(0, 0, 1, 0),
        vec4(vs_matrix3x2[1][1], vs_matrix3x2[1][2], 0, 1)
    );

    // vs_source_rectangle is the subtexture region to use
    // however, when the values are fetched they are normalized into a float in the range of [0,1] from an unsigned 16-bit value
    // thus, we need to multiply by 65535 to get the original unsigned 16-bit value as it was
    // the reason for using normalized values instead of just integers is for compatibility accross all graphics APIs
    vec2 src_size = vs_source_rectangle.zw * 65535;
    vec2 src_position = vs_source_rectangle.xy * 65535;

    // the corner position of the rectangle in model space
    vec2 corner = corners[gl_VertexIndex];
    // the vertex position in traditional sprite rendering
    vec4 position = vec4(corner * src_size, vs_depth, 1);
    // by convention we are using column-major matrices, thus the matrix multiplication is post-multiplication:
    // clip position = [View-To-Projection matrix]x[World-to-View matrix]x[Model-to-World matrix]*[Model vertex position]
    gl_Position = ViewProjectionMatrix * model_matrix * position;

    // corner is remapped from [-1, 1] to [0, 1] to be in texture coordinates space, bottom-left is origin
    vec2 corner_uv = corner * 0.5 + 0.5;
    // however, we want texture coordinates to use top-left origin because that's how people traditionally think of pixel space of an image
    // so we need to flip the Y-axis
    corner_uv.y = 1 - corner_uv.y;

    // calculate and pass texture coordinates to fragment shader
    fs_uv = (src_position + corner_uv * src_size) / TextureSize;

    // pass color to fragment shader
    fs_color = vs_color;
}

@end

@fs fs

uniform sampler2D Texture;

layout(location = 0) in vec4 fs_color;
layout(location = 1) in vec2 fs_uv;

out vec4 color;

void main()
{
    color = texture(Texture, fs_uv) * fs_color;
}

@end

@program spritebatch vs fs


make multi-factor authentication required for dsa aws buckets

If possible, we would like MFA required (rather than optional) for all new and current users for the following five aae AWS buckets:

aae-bppe-dev-us-west-2
aae-cdss-dev-us-west-2
aae-cdtfa-dev-us-west-2
aae-hcd-dev-us-west-2
aae-water-dev-us-west-2

This isn't urgent, but it would be ideal if this was done within the next week or two. If this isn't possible, could you provide us with a list of all current users with access to these five buckets so that we can confirm that they have MFA enabled.
Thanks!

@eaeslami I've been doing some more thinking and reading on this, and I think there probably is a way forward that would not be too onerous for data analysis projects, but would still require MFA (some resources linked in #128). It's not top of the priority list, but something we could probably do in the next few weeks.

interesting. yes, definitely not a top priority, but if we have time to discuss when you have more details that would be great!


Letter spacing is wrong for text with long length

Steps to reproduce

Create Text widget with text of length >= 4,449,555 characters.

Expected results
Letter spacing is correct and consistent throughout the entire text.
Actual results
Letter spacing is not consistent and is wrong towards the end of the text.
Code sample
Code sample
import 'package:flutter/material.dart';

void main() => runApp(const SampleApp());

class SampleApp extends StatelessWidget {
  const SampleApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: Scaffold(
        appBar: AppBar(title: const Text('Sample')),
        body: ListView(
          children: [
            Align(
              alignment: Alignment.topCenter,
              child: Container(
                width: 512,
                child: Text(
                  'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.' * 9999,
                ),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

Screenshots or Video

Screenshots / Video demonstration
Correct letter spacing at the beginning of the text:

Wrong letter spacing at the end of the text (this text is 4,449,555 characters long):

Logs
No response
Flutter Doctor output
Doctor output
[✓] Flutter (Channel stable, 3.27.1, on macOS 15.2 24C101 darwin-arm64, locale en-US)
    • Flutter version 3.27.1 on channel stable at /Users/charleslu/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 17025dd882 (4 days ago), 2024-12-17 03:23:09 +0900
    • Engine revision cb4b5fff73
    • Dart version 3.6.0
    • DevTools version 2.40.2

[✗] Android toolchain - develop for Android devices
    ✗ Unable to locate Android SDK.
      Install Android Studio from: https://developer.android.com/studio/index.html
      On first launch it will assist you in installing the Android SDK components.
      (or visit https://flutter.dev/to/macos-android-setup for detailed instructions).
      If the Android SDK has been installed to a custom location, please use
      `flutter config --android-sdk` to update to that location.

[✗] Xcode - develop for iOS and macOS
    ✗ Xcode installation is incomplete; a full installation is necessary for iOS and macOS development.
      Download at: https://developer.apple.com/xcode/
      Or install Xcode via the App Store.
      Once installed, run:
        sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
        sudo xcodebuild -runFirstLaunch
    ✗ CocoaPods not installed.
        CocoaPods is a package manager for iOS or macOS platform code.
        Without CocoaPods, plugins will not work on iOS or macOS.
        For more info, see https://flutter.dev/to/platform-plugins
      For installation instructions, see https://guides.cocoapods.org/using/getting-started.html#installation

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[!] Android Studio (not installed)
    • Android Studio not found; download from https://developer.android.com/studio/index.html
      (or visit https://flutter.dev/to/macos-android-setup for detailed instructions).

[✓] VS Code (version 1.96.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.102.0

[✓] Connected device (2 available)
    • macOS (desktop) • macos  • darwin-arm64   • macOS 15.2 24C101 darwin-arm64
    • Chrome (web)    • chrome • web-javascript • Google Chrome 131.0.6778.205

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 3 categories.

Reproducible using the code sample provided above.

flutter doctor -v
[✓] Flutter (Channel stable, 3.27.1, on macOS 15.1.1 24B91 darwin-arm64, locale en-US)
    • Flutter version 3.27.1 on channel stable at /Users/deanli/dev/stable
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 17025dd882 (7 days ago), 2024-12-17 03:23:09 +0900
    • Engine revision cb4b5fff73
    • Dart version 3.6.0
    • DevTools version 2.40.2

[!] Flutter (Channel master, 3.28.0-2.0.pre.38574, on macOS 15.1.1 24B91 darwin-arm64, locale en-US)
    • Flutter version 3.28.0-2.0.pre.38574 on channel master at /Users/deanli/dev/master
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 5c7d9d01ba (16 hours ago), 2024-12-22 10:11:21 -0800
    • Engine revision 5c7d9d01ba
    • Dart version 3.7.0 (build 3.7.0-267.0.dev)
    • DevTools version 2.41.0


Demo Fails To Do Anything

I visited the " >Working Demo<" in the readme file posted on this project, the link resolves to
 https://fossurl.palash.tk/

and entered "http://www.google.com" in the box and pushed the button "Short Me" and nothing happens, it is as if the button was nonfunctional.

I think it worked. Have you read the instructions below the button?
Whenever you shorten a url (enter an url in box and push the Short Me button)  , short url is automatically copied to your clipboard

No, I did not read the instructions below the button.
I tried entering "http://www.google.com" and then found in my clipboard
"https://fossurl.palash.tk/#gc3foi"
Thank you. I'm closing this ticket.

Happy To Hear


Add qualifier for documented location of ad://ext/AuthenticationSilo

All the other objects listed mention this fact (e.g. create in the Configuration partition), so this just makes it consistent.

#sign-off


Feature Request: Automatically display journal entries using a pager (like less)

It would be great if there were a jrnl setting that sets it to automatically pipe jrnl entries to less when displaying them. Then I wouldn't have to clear my entire terminal output after just reading a past entry.

jrnl is meant to play nicely with other command line tools, so I'd recommend just creating a bash alias for that, like alias lrnl="jrnl -n 1 | less"


Add new params to add RouteSettings to Navigator

The default MaterialPageRoute take settings (RouteSettings) as param, is neccesary to add a route in the Navigator. But  we can' t pass the settings as param to the constructor PageTransition

Alternatively, extract the arguments using onGenerateRoute;
Instead of extracting the arguments directly inside the widget, you can also extract the arguments inside an onGenerateRoute() function and pass them to a widget.
onGenerateRoute: (settings) {
  switch (settings.name) {
    case '/second':
      return PageTransition(child: SecondPage(), type: PageTransitionType.scale);
      break;
    default:
      return null;
  }
},


using --error-reporting alone sends a useless report

Description
Maybe there's no point in sending an empty sentry report when someone simply enables error reporting for testing
$ driftctl --error-reporting
Detect, track and alert on infrastructure drift.

Usage: driftctl <command> [flags]

COMMANDS:
  completion               Generate completion script
  scan                     Scan
  version                  Display driftctl version

FLAGS:
      --error-reporting    Enable error reporting.
                           WARNING: may leak sensitive data
  -h, --help               Display help for command
      --no-version-check   Disable the version check

LEARN MORE:
  Use "driftctl <command> --help" for more information about a command
Sending error report ... done, thank you ❤

Environment

OS: fedora 33
driftctl version: 0.7.0

How to reproduce

Possible Solution
Maybe do not send the report automatically if there´s nothing to send, or no command used
Additional context

@sjourdan Nothing is really sent if no crash happened.
We cannot know if something was captured by sentry, so maybe we can remove this message ?

Actually, with the v0.7.1 this flag behaviour is different.

Actually, with the v0.7.1 this flag behaviour is different.

The flag name has changed but the behavior remain the same

Actually, with the v0.7.1 this flag behaviour is different.

The flag name has changed but the behavior remain the same

`./bin/driftctl_darwin_amd64 --send-crash-report
Detect, track and alert on infrastructure drift.
Usage: driftctl  [flags]
COMMANDS:
completion               Generate completion script
scan                     Scan
version                  Display driftctl version
FLAGS:
-h, --help                Display help for command
--no-version-check    Disable the version check
--send-crash-report   Enable error reporting. Crash data will be sent to us via Sentry.
WARNING: may leak sensitive data (please read documentation for more details)
This flag should be used only if an error occur during execution
LEARN MORE:
Use "driftctl  --help" for more information about a command`
For example i'm talking about the message in your previous discussion with sjourdan

This is a bug that will be fixed with https://github.com/cloudskiff/driftctl/pull/424

@sjourdan Nothing is really sent if no crash happened.
We cannot know if something was captured by sentry, so maybe we can remove this message ?

@eliecharra May be we could replace this with a debug message ? I'll push a PR about that.


Clean up unused classes

There are a few packages that aren't used currently and could cause user confusion.
org.springframework.cloud.gcp.pubsub.converters.* (except SimpleMessageConverter)
org.springframework.cloud.gcp.pubsub.converters.support
org.springframework.cloud.gcp.pubsub.*
We should remove them before launching anything to the public.
They cover areas that we already have issues for: (a)sync mode (#51) and topic/message conversion (#54). We should wait until those issues are worked on in order to provide those features. Until then, it's better to remove the classes so they don't confuse users.
@artembilan wdyt?

I don't think so that it is a time to remove something.
Let's come back to this when we will be on RC at least!
Any Milestone assumes some breaking changes anyway.

We're really close to an alpha release -- only three issues left to resolve, which are already half-resolved.
We would like to launch an alpha version of the channel adapters + storage resource abstraction sooner rather than later. Feel free to add anything to the milestone if you think it's a blocker.
I don't think we should release the alpha with those unused classes, but I also don't think that those issues are alpha blockers, hence the suggestion to remove them for the time being (and possibly add them again later).

OK. I can't answer to this question. I don't manage this project.
I only review the code for quality and share some ideas if I can.
All the management steps our outside of my responsibility.
@markfisher , @viniciusccarvalho , who manages this project?
Who makes decision what and when should be released?
Thanks

I think the release plans is something we can discuss in the next meeting.
The question here is whether it's OK to remove these classes for now. We can always bring them back if the need arises.

Those classes can all be removed. It was part of a pilot on getting a reactive support for pubsub binder.
After our discussions with the pubsub team, I'm not really convinced of a pubsub binder at all ATM. Making more sense as a SI and Source/Sink for stream right now.
Until we revisit the binder API for 2.0, I'd say there's little benefit on trying to get a pubsub binder

I took the liberty to open a PR to address this issue, to be merged when we think it's the right time.
I would personally like to see these classes removed sooner rather than later so that snapshot users don't get sidetracked by classes that don't work. The getting started guide should be almost ready and snapshots are now being published so, in practice, we could start seeing some users now.


WIP: BLD: fix boost Cython compilation with Meson

This gets rid of run_command, and replaces it with use of custom_target. This means things now run at build-time rather than at configure-time, and cache will work correctly.
Closes #25
scipy/optimize/_highs/meson.build is still TODO.
This was remarkably painful, mostly because of the Cython bug in the generated code that's fixed in the first commit.

mostly because of the Cython bug in the generated code that's fixed in the first commit.

I looked into it some more. How relative/absolute imports in Cython works is badly documented. The original setup.py code was working, because the generated code went into a newly created directory _boost/src/ which did not have an __init__.py file. At that point it looked to Cython like it was a set of standalone .pyx/.pxd files not part of any package. So an absolute import succeeded.
After copying those exact same .pyx/.pxd files to the build directory, the absolute import failed, because there was a __init__.py file there. Changing it to a relative import then fixed the problem.
The tests don't actually run yet (true for most submodules), so once they do I expect some more problems to surface.

Okay, _highs was straightforward - no codegen involved, so just replacing run_command with custom_target did the job.
@czgdp1807 note that I also reformatted the file. The style guide for Meson says indentation must be 2 spaces, and I've tried to stick to that. Not a big deal now, but I think it'd be nice to go back and fix that up in one go before merging things into SciPy master.


[Code Generation] Use JSON file relative directory for the generated class package

Like Sqldelight does

this issue on me


how to get url for youtube

how i can give url to other person??
how get url from key???

I don't understand what do you mean, but you should find in youtube documentation or internet. This is only for issues related with the library.

@pedroSG94
ok no problem
please ans in this please
https://github.com/pedroSG94/rtmp-rtsp-stream-client-java/issues/705


Dev/databackendretake

Implement a data file backend.
Functional tests pass.
This assumes that /data/pathNumbers exist on machine.  Do we want to create that through the Docker image and instruct in the readme or handle some other way?

Ready for final reviews.  Thanks, @RachedBenMustapha for all of the heavy lifting on the file flags!
I can drop the changes to the metadata wrapper depending on which PR gets merged first (this or the metadata file backend).
Since the ReadMe for master has diverged, I will update the readMe separately.

@ironman-machine any concerns?

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow http://ci.ironmann.io/gh/scality/Integration/1609 for circle status.

A few minor comments/questions, but I'm 👍 .

let's use localData and localMetadata names.

👍

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow http://ci.ironmann.io/gh/scality/Integration/1613 for circle status.

@ironman-machine SVC_BASE_FORCE_BUILD=1
NODESVC_BASE_FORCE_BUILD=1

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"@IRONMAN-MACHINE SVC_BASE_FORCE_BUILD":"1","NODESVC_BASE_FORCE_BUILD":"1","DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow the last build of wip%2Fultron for circle status.

@ironman-machine SVC_BASE_FORCE_BUILD=1
NODESVC_BASE_FORCE_BUILD=1

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"@IRONMAN-MACHINE SVC_BASE_FORCE_BUILD":"1","NODESVC_BASE_FORCE_BUILD":"1","DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow the last build of wip%2Fultron for circle status.

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"@IRONMAN-MACHINE SVC_BASE_FORCE_BUILD":"1","NODESVC_BASE_FORCE_BUILD":"1","DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow the last build of wip%2Fultron for circle status.

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"@IRONMAN-MACHINE SVC_BASE_FORCE_BUILD":"1","NODESVC_BASE_FORCE_BUILD":"1","DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow the last build of wip%2Fultron for circle status.

@ironman-machine where are you
SVC_BASE_FORCE_BUILD=1
NODESVC_BASE_FORCE_BUILD=1

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"SVC_BASE_FORCE_BUILD":"1","NODESVC_BASE_FORCE_BUILD":"1","DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow http://ci.ironmann.io/gh/scality/Integration/1616 for circle status.

👍

@ironman-machine get to it

Hello @LaurenSpiegel,
Starting end to end procedure using the following payload:
{"DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow http://ci.ironmann.io/gh/scality/Integration/1619 for circle status.

@ironman-machine No sleep for the warriors, go and charge !

Hello @DavidPineauScality,
Starting end to end procedure using the following payload:
{"DEFAULT_BRANCH":"rel/1.1","SCALITY_S3_BRANCH":"dev/databackendretake"}
Please follow http://ci.ironmann.io/gh/scality/Integration/1622 for circle status.

Merging on behalf of @LaurenSpiegel since End-to-End passes :) Congrats ;)


[Kafka-ws] Kafka-ws flaps between Ready and NotReady after redis connection be broken and reestablished

I'm submitting a ...

[X] bug report
[ ] feature request
[ ] support request

Do you want to request a feature or report a bug?

What is the current behavior?

Kafka-ws flaps between Ready and NotReady after redis connection be broken and reestablished. Then, it remains not ready.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

1-Interrupt connection between kafka-ws and kafka-ws-redis
2-try to connect a WS client.
3-wait until kafka-ws becomes not ready
4-reestablish the connection between kafka-ws and kafka-ws-redis
5-wait until kafka-ws becomes ready
6-connect a WS client. (connection is established successfully)
7-see that kafka-ws becomes not ready

Logs and captures:
rc3_redis_connection.zip

What is the expected behavior?
after step 5, kafka-ws shall keep ready.

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Version: 0.6.0-rc3
Environment: kubernetes
Operating system: Ubuntu 18.04

Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc)

steps to reproduce the problem from @ctossato:
docker exec dojot_redis_1 ip link show

eth0@if371: ....

ip a | grep 371

sudo ip link set vethe4d074a down

I added a heathchecker that ping/pong the redis, but I noticed that it didn't work properly using the steps above because the service never received any error as an answer, it was stopped.
The solution was to add a timeout in case it was not possible to obtain the pong.
I don't know if it's the best solution.

stop kafka-ws-redis

kafka-ws_1                    | 2021-05-06T17:31:37.182Z -- kafka-ws:redis-expire-mgmt -- info: sub: onEnd
kafka-ws_1                    | 2021-05-06T17:31:37.182Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":1,"error":null,"total_retry_time":0,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:31:37.183Z -- kafka-ws:redis-expire-mgmt -- info: pub: onEnd
kafka-ws_1                    | 2021-05-06T17:31:37.183Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":1,"error":null,"total_retry_time":0,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:31:37.183Z -- kafka-ws:redis-manager -- info: end
kafka-ws_1                    | 2021-05-06T17:31:37.183Z -- kafka-ws:redis-manager -- debug: Retry strategy options { attempt: 1, error: null, total_retry_time: 0, times_connected: 2 }
kafka-ws_1                    | 2021-05-06T17:31:42.184Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:31:42.184Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:31:42.185Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:31:47.186Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":2,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":5000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:31:47.187Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":2,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":5000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:31:52.189Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:31:52.190Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:31:52.192Z -- kafka-ws:redis-manager -- debug: Retry strategy options {
kafka-ws_1                    |   attempt: 2,
kafka-ws_1                    |   error: Error: Redis connection to kafka-ws-redis:6379 failed - getaddrinfo EAI_AGAIN kafka-ws-redis
kafka-ws_1                    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
kafka-ws_1                    |     errno: 'EAI_AGAIN',
kafka-ws_1                    |     code: 'EAI_AGAIN',
kafka-ws_1                    |     syscall: 'getaddrinfo',
kafka-ws_1                    |     hostname: 'kafka-ws-redis'
kafka-ws_1                    |   },
kafka-ws_1                    |   total_retry_time: 5000,
kafka-ws_1                    |   times_connected: 2
kafka-ws_1                    | }
kafka-ws_1                    | 2021-05-06T17:31:55.473Z -- kafka-ws:redis-manager -- debug: Checking if redis is health...
kafka-ws_1                    | 2021-05-06T17:31:55.490Z -- kafka-ws:redis-expire-mgmt -- debug: Checking if redis-expire-pub is health...
kafka-ws_1                    | 2021-05-06T17:31:55.492Z -- kafka-ws:redis-expire-mgmt -- debug: Checking if redis-expire-sub is health...
kafka-ws_1                    | 2021-05-06T17:31:55.505Z -- kafka-ws:kafka-consumer -- debug: Kafka is health
kafka-ws_1                    | 2021-05-06T17:31:57.193Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:31:57.193Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":3,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":10000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:31:57.194Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":3,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":10000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:32:00.477Z -- kafka-ws:redis-manager -- warn: redis is not health because time exceeded to receive pong.
kafka-ws_1                    | 2021-05-06T17:32:00.478Z -- kafka-ws:redis-manager -- error: There was a problem trying to check if the redis is healthy. e=time exceeded to receive pong {"stack":"Error: time exceeded to receive pong\n    at Timeout._onTimeout (/opt/kafka-ws/app/Redis/Utils.js:25:25)\n    at listOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)"}
kafka-ws_1                    | 2021-05-06T17:32:00.490Z -- kafka-ws:redis-expire-mgmt -- warn: redis-expire-pub is not health because time exceeded to receive pong.
kafka-ws_1                    | 2021-05-06T17:32:00.491Z -- kafka-ws:redis-expire-mgmt -- error: There was a problem trying to check if the redis-expire-pub is healthy. e=time exceeded to receive pong {"stack":"Error: time exceeded to receive pong\n    at Timeout._onTimeout (/opt/kafka-ws/app/Redis/Utils.js:25:25)\n    at listOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)"}
kafka-ws_1                    | 2021-05-06T17:32:00.493Z -- kafka-ws:redis-expire-mgmt -- warn: redis-expire-sub is not health because time exceeded to receive pong.
kafka-ws_1                    | 2021-05-06T17:32:00.494Z -- kafka-ws:redis-expire-mgmt -- error: There was a problem trying to check if the redis-expire-sub is healthy. e=time exceeded to receive pong {"stack":"Error: time exceeded to receive pong\n    at Timeout._onTimeout (/opt/kafka-ws/app/Redis/Utils.js:25:25)\n    at listOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)"}
kafka-ws_1                    | 2021-05-06T17:32:02.196Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:02.197Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:02.201Z -- kafka-ws:redis-manager -- debug: Retry strategy options {
kafka-ws_1                    |   attempt: 3,
kafka-ws_1                    |   error: Error: Redis connection to kafka-ws-redis:6379 failed - getaddrinfo EAI_AGAIN kafka-ws-redis
kafka-ws_1                    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
kafka-ws_1                    |     errno: 'EAI_AGAIN',
kafka-ws_1                    |     code: 'EAI_AGAIN',
kafka-ws_1                    |     syscall: 'getaddrinfo',
kafka-ws_1                    |     hostname: 'kafka-ws-redis'
kafka-ws_1                    |   },
kafka-ws_1                    |   total_retry_time: 10000,
kafka-ws_1                    |   times_connected: 2
kafka-ws_1                    | }
kafka-ws_1                    | 2021-05-06T17:32:07.202Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":4,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":15000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:32:07.202Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":4,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":15000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:32:07.203Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:12.204Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:12.205Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:12.210Z -- kafka-ws:redis-manager -- debug: Retry strategy options {
kafka-ws_1                    |   attempt: 4,
kafka-ws_1                    |   error: Error: Redis connection to kafka-ws-redis:6379 failed - getaddrinfo EAI_AGAIN kafka-ws-redis
kafka-ws_1                    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
kafka-ws_1                    |     errno: 'EAI_AGAIN',
kafka-ws_1                    |     code: 'EAI_AGAIN',
kafka-ws_1                    |     syscall: 'getaddrinfo',
kafka-ws_1                    |     hostname: 'kafka-ws-redis'
kafka-ws_1                    |   },
kafka-ws_1                    |   total_retry_time: 15000,
kafka-ws_1                    |   times_connected: 2
kafka-ws_1                    | }
kafka-ws_1                    | 2021-05-06T17:32:17.210Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":5,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":20000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:32:17.210Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:17.212Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":5,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":20000,"times_connected":2}
kafka-ws_1                    | 2021-05-06T17:32:22.211Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:22.213Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:32:22.218Z -- kafka-ws:redis-manager -- debug: Retry strategy options {
kafka-ws_1                    |   attempt: 5,
kafka-ws_1                    |   error: Error: Redis connection to kafka-ws-redis:6379 failed - getaddrinfo EAI_AGAIN kafka-ws-redis
kafka-ws_1                    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
kafka-ws_1                    |     errno: 'EAI_AGAIN',
kafka-ws_1                    |     code: 'EAI_AGAIN',
kafka-ws_1                    |     syscall: 'getaddrinfo',
kafka-ws_1                    |     hostname: 'kafka-ws-redis'
kafka-ws_1                    |   },
kafka-ws_1                    |   total_retry_time: 20000,
kafka-ws_1                    |   times_connected: 2
kafka-ws_1                    | }

connecting a WS client.

kafka-ws_1                    | 2021-05-06T17:37:37.782Z -- kafka-ws:redis-expire-mgmt -- info: pub: onEnd
kafka-ws_1                    | 2021-05-06T17:37:37.782Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":1,"error":null,"total_retry_time":0,"times_connected":3}
kafka-ws_1                    | 2021-05-06T17:37:37.783Z -- kafka-ws:redis-expire-mgmt -- info: sub: onEnd
kafka-ws_1                    | 2021-05-06T17:37:37.784Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":1,"error":null,"total_retry_time":0,"times_connected":3}
kafka-ws_1                    | 2021-05-06T17:37:37.784Z -- kafka-ws:redis-manager -- info: end
kafka-ws_1                    | 2021-05-06T17:37:37.785Z -- kafka-ws:redis-manager -- debug: Retry strategy options { attempt: 1, error: null, total_retry_time: 0, times_connected: 3 }
kafka-ws_1                    | 2021-05-06T17:37:42.785Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:37:42.786Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:37:42.788Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:37:45.604Z -- kafka-ws:app -- error: The application is not in a ready state, the request cannot be handled
kafka-ws_1                    | 2021-05-06T17:37:45.604Z -- kafka-ws:app -- debug: ServiceUnavailableError: Service Unavailable
kafka-ws_1                    |     at /opt/kafka-ws/app/App.js:40:14
kafka-ws_1                    |     at newFn (/opt/kafka-ws/node_modules/express-async-errors/index.js:16:20)
kafka-ws_1                    |     at Layer.handle [as handle_request] (/opt/kafka-ws/node_modules/express/lib/router/layer.js:95:5)
kafka-ws_1                    |     at trim_prefix (/opt/kafka-ws/node_modules/express/lib/router/index.js:317:13)
kafka-ws_1                    |     at /opt/kafka-ws/node_modules/express/lib/router/index.js:284:7
kafka-ws_1                    |     at Function.process_params (/opt/kafka-ws/node_modules/express/lib/router/index.js:335:12)
kafka-ws_1                    |     at next (/opt/kafka-ws/node_modules/express/lib/router/index.js:275:10)
kafka-ws_1                    |     at expressInit (/opt/kafka-ws/node_modules/express/lib/middleware/init.js:40:5)
kafka-ws_1                    |     at newFn (/opt/kafka-ws/node_modules/express-async-errors/index.js:16:20)
kafka-ws_1                    |     at Layer.handle [as handle_request] (/opt/kafka-ws/node_modules/express/lib/router/layer.js:95:5)
kafka-ws_1                    | 2021-05-06T17:37:47.789Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":2,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":5000,"times_connected":3}
kafka-ws_1                    | 2021-05-06T17:37:47.790Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":2,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":5000,"times_connected":3}

kafka-ws unhealthy

start kafka-ws-redis

kafka-ws_1                    | 2021-05-06T17:38:30.505Z -- kafka-ws:redis-manager -- warn: redis is not health because time exceeded to receive pong.
kafka-ws_1                    | 2021-05-06T17:38:30.505Z -- kafka-ws:redis-manager -- error: There was a problem trying to check if the redis is healthy. e=time exceeded to receive pong {"stack":"Error: time exceeded to receive pong\n    at Timeout._onTimeout (/opt/kafka-ws/app/Redis/Utils.js:25:25)\n    at listOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)"}
kafka-ws_1                    | 2021-05-06T17:38:30.505Z -- kafka-ws:redis-expire-mgmt -- warn: redis-expire-pub is not health because time exceeded to receive pong.
kafka-ws_1                    | 2021-05-06T17:38:30.506Z -- kafka-ws:redis-expire-mgmt -- error: There was a problem trying to check if the redis-expire-pub is healthy. e=time exceeded to receive pong {"stack":"Error: time exceeded to receive pong\n    at Timeout._onTimeout (/opt/kafka-ws/app/Redis/Utils.js:25:25)\n    at listOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)"}
kafka-ws_1                    | 2021-05-06T17:38:30.506Z -- kafka-ws:redis-expire-mgmt -- warn: redis-expire-sub is not health because time exceeded to receive pong.
kafka-ws_1                    | 2021-05-06T17:38:30.506Z -- kafka-ws:redis-expire-mgmt -- error: There was a problem trying to check if the redis-expire-sub is healthy. e=time exceeded to receive pong {"stack":"Error: time exceeded to receive pong\n    at Timeout._onTimeout (/opt/kafka-ws/app/Redis/Utils.js:25:25)\n    at listOnTimeout (internal/timers.js:549:17)\n    at processTimers (internal/timers.js:492:7)"}

kafka-ws_1                    | 2021-05-06T17:38:32.816Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:38:32.817Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:38:32.819Z -- kafka-ws:redis-manager -- debug: Retry strategy options {
kafka-ws_1                    |   attempt: 6,
kafka-ws_1                    |   error: Error: Redis connection to kafka-ws-redis:6379 failed - getaddrinfo EAI_AGAIN kafka-ws-redis
kafka-ws_1                    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
kafka-ws_1                    |     errno: 'EAI_AGAIN',
kafka-ws_1                    |     code: 'EAI_AGAIN',
kafka-ws_1                    |     syscall: 'getaddrinfo',
kafka-ws_1                    |     hostname: 'kafka-ws-redis'
kafka-ws_1                    |   },
kafka-ws_1                    |   total_retry_time: 25000,
kafka-ws_1                    |   times_connected: 3
kafka-ws_1                    | }

kafka-ws healthy

kafka-ws_1                    | 2021-05-06T17:40:47.904Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":20,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":95000,"times_connected":3}
kafka-ws_1                    | 2021-05-06T17:40:47.904Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:40:47.905Z -- kafka-ws:redis-expire-mgmt -- debug: Retry strategy options {"attempt":20,"error":{"errno":"EAI_AGAIN","code":"EAI_AGAIN","syscall":"getaddrinfo","hostname":"kafka-ws-redis"},"total_retry_time":95000,"times_connected":3}
kafka-ws_1                    | 2021-05-06T17:40:52.906Z -- kafka-ws:redis-expire-mgmt -- warn: pub: reconnecting
kafka-ws_1                    | 2021-05-06T17:40:52.907Z -- kafka-ws:redis-expire-mgmt -- warn: sub: reconnecting
kafka-ws_1                    | 2021-05-06T17:40:52.909Z -- kafka-ws:redis-manager -- debug: Retry strategy options {
kafka-ws_1                    |   attempt: 20,
kafka-ws_1                    |   error: Error: Redis connection to kafka-ws-redis:6379 failed - getaddrinfo EAI_AGAIN kafka-ws-redis
kafka-ws_1                    |       at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:66:26) {
kafka-ws_1                    |     errno: 'EAI_AGAIN',
kafka-ws_1                    |     code: 'EAI_AGAIN',
kafka-ws_1                    |     syscall: 'getaddrinfo',
kafka-ws_1                    |     hostname: 'kafka-ws-redis'
kafka-ws_1                    |   },
kafka-ws_1                    |   total_retry_time: 95000,
kafka-ws_1                    |   times_connected: 3
kafka-ws_1                    | }
kafka-ws_1                    | 2021-05-06T17:40:53.926Z -- kafka-ws:redis-expire-mgmt -- info: pub: Connect
kafka-ws_1                    | 2021-05-06T17:40:53.927Z -- kafka-ws:redis-expire-mgmt -- debug: sub: ready.
kafka-ws_1                    | 2021-05-06T17:40:53.927Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.927Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.928Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.928Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.928Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.928Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.928Z -- kafka-ws:redis-expire-mgmt -- info: Subscribed to __keyevent@1__:expired event channel
kafka-ws_1                    | 2021-05-06T17:40:53.928Z -- kafka-ws:redis-expire-mgmt -- info: sub: Connect
kafka-ws_1                    | 2021-05-06T17:40:53.929Z -- kafka-ws:redis-expire-mgmt -- debug: pub: ready.
kafka-ws_1                    | 2021-05-06T17:40:53.929Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.930Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.930Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.930Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.930Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:53.930Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:55.511Z -- kafka-ws:redis-manager -- debug: Checking if redis is health...
kafka-ws_1                    | 2021-05-06T17:40:55.512Z -- kafka-ws:redis-expire-mgmt -- debug: Checking if redis-expire-pub is health...
kafka-ws_1                    | 2021-05-06T17:40:55.513Z -- kafka-ws:redis-expire-mgmt -- debug: Checking if redis-expire-sub is health...
kafka-ws_1                    | 2021-05-06T17:40:55.514Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:40:55.515Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:40:55.516Z -- kafka-ws:kafka-consumer -- debug: Kafka is health
kafka-ws_1                    | 2021-05-06T17:40:57.912Z -- kafka-ws:redis-manager -- warn: reconnecting
kafka-ws_1                    | 2021-05-06T17:40:57.913Z -- kafka-ws:redis-manager -- info: connect
kafka-ws_1                    | 2021-05-06T17:40:57.913Z -- kafka-ws:redis-manager -- debug: ready.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:40:57.914Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:41:25.513Z -- kafka-ws:redis-manager -- debug: Checking if redis is health...
kafka-ws_1                    | 2021-05-06T17:41:25.515Z -- kafka-ws:redis-expire-mgmt -- debug: Checking if redis-expire-pub is health...
kafka-ws_1                    | 2021-05-06T17:41:25.516Z -- kafka-ws:redis-expire-mgmt -- debug: Checking if redis-expire-sub is health...
kafka-ws_1                    | 2021-05-06T17:41:25.517Z -- kafka-ws:redis-manager -- debug: redis is health.
kafka-ws_1                    | 2021-05-06T17:41:25.518Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-pub is health.
kafka-ws_1                    | 2021-05-06T17:41:25.518Z -- kafka-ws:redis-expire-mgmt -- debug: redis-expire-sub is health.
kafka-ws_1                    | 2021-05-06T17:41:25.520Z -- kafka-ws:kafka-consumer -- debug: Kafka is health

connecting a WS client.

kafka-ws_1                    | 2021-05-06T17:44:27.192Z -- kafka-ws:app -- info: <IP_ADDRESS> - - a7baa377-ec61-42b5-840a-f0d6005dfdde 'GET /kafka-ws/v1/ticket HTTP/1.1' 200 77 - python-requests/2.25.1
kafka-ws_1                    |
kafka-ws_1                    | 2021-05-06T17:44:27.198Z -- kafka-ws:websocket-tarball -- debug: Received connection from <IP_ADDRESS>:51776
kafka-ws_1                    | 2021-05-06T17:44:27.198Z -- kafka-ws:websocket-tarball -- debug: Setting expiration connection to<PHONE_NUMBER> sec
kafka-ws_1                    | 2021-05-06T17:44:27.198Z -- kafka-ws:redis-expire-mgmt -- debug: addConnection: 6323994a-61d7-4304-b1d4-9e96e95b0356<PHONE_NUMBER>
kafka-ws_1                    | 2021-05-06T17:44:27.199Z -- kafka-ws:kafka-topics-consumer-callbacks-mgmt -- debug: addCallbackToTopic: Topic=admin.device-data idCallback=6323994a-61d7-4304-b1d4-9e96e95b0356
kafka-ws_1                    | 2021-05-06T17:44:27.199Z -- kafka-ws:kafka-consumer -- debug: registerCallback: Register Callback for topic admin.device-data
kafka-ws_1                    | 2021-05-06T17:44:27.199Z -- microservice-sdk:consumer -- debug: subscribing on topic: admin.device-data
kafka-ws_1                    | 2021-05-06T17:44:27.199Z -- microservice-sdk:consumer -- debug: Refreshing subscriptions
kafka-ws_1                    | 2021-05-06T17:44:27.199Z -- microservice-sdk:consumer -- debug: subscribing in the following topics (1): ["admin.device-data"]
kafka-ws_1                    | 2021-05-06T17:44:27.200Z -- kafka-ws:kafka-consumer -- debug: registerCallback: All Registered Callbacks admin.device-data,2a218a7b-eaa7-41e0-b93c-b5b5e45ba965
kafka-ws_1                    | 2021-05-06T17:44:27.203Z -- microservice-sdk:consumer -- debug: Current queue utilization:
kafka-ws_1                    |     (97 + 0)/10485760 bytes
kafka-ws_1                    | 2021-05-06T17:44:27.204Z -- microservice-sdk:consumer -- debug: Message on topic: admin.device-data. Calling callback: 2a218a7b-eaa7-41e0-b93c-b5b5e45ba965. Attempt: 1/0
kafka-ws_1                    | 2021-05-06T17:44:27.204Z -- kafka-ws:kafka-topics-consumer-callbacks-mgmt -- debug: createCallbackKafka: Topic=admin.device-data Payload={
kafka-ws_1                    |   metadata: { deviceid: '5fe63f', tenant: 'admin', timestamp:<PHONE_NUMBER>400 },
kafka-ws_1                    |   attrs: { int: 883 }
kafka-ws_1                    | }
kafka-ws_1                    | 2021-05-06T17:44:27.206Z -- kafka-ws:websocket-tarball -- debug: Sending {"attrs":{"int":692}}
kafka-ws_1                    | 2021-05-06T17:44:27.206Z -- microservice-sdk:consumer -- debug: Calling resume consumer; isPaused? false...
kafka-ws_1                    | 2021-05-06T17:44:27.206Z -- microservice-sdk:consumer -- debug: Current queue utilization:
kafka-ws_1                    |     (97 + 0)/10485760 bytes
kafka-ws_1                    | 2021-05-06T17:44:27.228Z -- microservice-sdk:consumer -- debug: Message on topic: admin.device-data. Calling callback: 2a218a7b-eaa7-41e0-b93c-b5b5e45ba965. Attempt: 1/0
kafka-ws_1                    | 2021-05-06T17:44:27.229Z -- kafka-ws:kafka-topics-consumer-callbacks-mgmt -- debug: createCallbackKafka: Topic=admin.device-data Payload={
kafka-ws_1                    |   metadata: { deviceid: '5fe63f', tenant: 'admin', timestamp:<PHONE_NUMBER>458 },
kafka-ws_1                    |   attrs: { int: 828 }
kafka-ws_1                    | }


cannot import name 'QhullError' from 'scipy.spatial'

Describe your issue.
The version I installed is 1.5.0. What is the reason for this issue? Can you help me answer it? Thank you
Reproducing Code Example
def compute_2d_bbox_xywh_from_pose(points, pose, K, width=640, height=480, clip=False):
Traceback (most recent call last):
  File "core/self6dpp/main_self6dpp.py", line 36, in <module>
    from core.self6dpp.datasets.dataset_factory import register_datasets_in_cfg
  File "/home/zhumengjie/self6dpp/core/self6dpp/../../core/self6dpp/datasets/dataset_factory.py", line 11, in <module>
    from core.self6dpp.datasets import (
  File "/home/zhumengjie/self6dpp/core/self6dpp/../../core/self6dpp/datasets/lm_dataset_d2.py", line 22, in <module>
    from lib.utils.mask_utils import binary_mask_to_rle, cocosegm2mask
  File "/home/zhumengjie/self6dpp/core/self6dpp/../../lib/utils/mask_utils.py", line 12, in <module>
    from skimage.morphology import binary_dilation as _binary_dilation_sk
  File "/home/zhumengjie/anaconda3/envs/self6d/lib/python3.8/site-packages/skimage/morphology/__init__.py", line 12, in <module>
    from .convex_hull import convex_hull_image, convex_hull_object
  File "/home/zhumengjie/anaconda3/envs/self6d/lib/python3.8/site-packages/skimage/morphology/convex_hull.py", line 4, in <module>
    from scipy.spatial import ConvexHull, QhullError
ImportError: cannot import name 'QhullError' from 'scipy.spatial' (/home/zhumengjie/anaconda3/envs/self6d/lib/python3.8/site-packages/scipy/spatial/__init__.py)

Error message
cannot import name 'QhullError' from 'scipy.spatial'

SciPy/NumPy/Python version and system information
>>> import sys, scipy, numpy
>>> print(scipy.__version__, numpy.__version__, sys.version_info)
1.5.0 1.23.0 sys.version_info(major=3, minor=8, micro=0, releaselevel='final', serial=0)
>>> scipy.show_config()
lapack_mkl_info:
  NOT AVAILABLE
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_mkl_info:
  NOT AVAILABLE
blis_info:
  NOT AVAILABLE
openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
>>>

Hi @onesmallwhite, it looks like QhullError is only exposed in the scipy.spatial namespace from SciPy 1.8 onwards. So you won't be able to import it this way in SciPy 1.5.

Your traceback includes a scikit-image call to skimage.morphology. Do you know which version of scikit-image you are using? From a quick look, I think that the most version to support SciPy 1.5 is v0.19. v0.20 and beyond require SciPy >= 1.8

Thanks for the answers here @lucascolley! They look correct to me, so this is a case of version mismatching. Upgrading SciPy should make the problem go away. There's no bug here, so I'll close the issue.


[SPARK-24547][K8S] Allow for building spark on k8s docker images without cache and don't forget to push spark-py container.

What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/SPARK-24547
TL;DR from JIRA issue:

First time I generated images for 2.4.0 Docker was using it's cache, so actually when running jobs, old jars where still in the Docker image. This produces errors like this in the executors:

java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID =<PHONE_NUMBER>931972169, local class serialVersionUID = -3720498261147521051

The second problem was that the spark container is pushed, but the spark-py container wasn't yet. This was just forgotten in the initial PR.

A third problem I also ran into because I had an older docker was https://github.com/apache/spark/pull/21551 so I have not included a fix for that in this ticket.

How was this patch tested?
I've tested it on my own Spark on k8s deployment.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Can one of the admins verify this patch?

can you edit the title to add [k8s] and fix the text which is truncated?

ok to test

Test build #91966 has started for PR 21555 at commit c17179b.

Test build #91966 has finished for PR 21555 at commit c17179b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91966/
Test PASSed.

retest this please

Test build #92054 has started for PR 21555 at commit c17179b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/299/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/299/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/299/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4193/
Test PASSed.

retest this please

Test build #92063 has started for PR 21555 at commit c17179b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/307/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4201/
Test PASSed.

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/307/

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/307/
Test FAILed.

Test build #92054 has finished for PR 21555 at commit c17179b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92054/
Test PASSed.

Test build #92063 has finished for PR 21555 at commit c17179b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92063/
Test PASSed.

retest this please

Test build #92103 has started for PR 21555 at commit c17179b.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/330/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/4224/
Test PASSed.

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/330/

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/330/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92103/
Test PASSed.

Good to merge with tests success. @mccheah and @foxish please merge

Lgtm. Will merge when tests pass. Thanks!

This change makes it such that using the tool forces building and pushing both Python and non-Python, but, what if the user wants to only build one to save time? I can imagine that being the case in something like a dev-CI workflow. Ideally the tool would allow one to be selective in managing either image or both.

@mccheah Good note. As this is a blocker for other PRs. It is probably best to refactor the docker-image-tool.sh in a separate PR for that is the not the focus of this PR.

@mccheah, that is a good point but I agree with @ifilonenko that we can do it in a subsequent PR. I'm thinking we merge this as-is and I can try to get a follow-up PR here for dockerfile refactor.

LGMT, I am OK to merge.
Most of the automated image build tooling I've seen is custom, but I agree w/ Matt that being able to selectively build is worth supporting, via a followup PR. It might make it easier to write custom logic using this as an abstraction.

Yup feel free to merge and follow up separately.

Will follow-up. Thanks all for the comments! Merging to master.


How to run one step adaptive inference or two step adaptive inference?

As I questioned in the title, I've tried default, filter and weight, upperbound inference, and they all worked well.
But I can't ran one step adaptive inference or two step adaptive inference just by change the inference parameter "--inference_style". Each time I ran it, the function "_fill_multi_content" in the "prompt.py" document always went wrong.
Is there any other parameters or code I need to change? I would sincerely appreciate your help!

Hi, thank you for submitting the issue. I have fully investigated the issue, and it turns out there's a bug with the prompt class for the two adaptive inference methods. We recently changed the way we handle nested prompts, and we apologize for not realizing this issue sooner.
I have made hot fixes to both the master branch of this repository and the datasets' repository (I had to change the prompt template for adaptive_info_prompt). Please update both repositories and see if it works.
We are undergoing some major updates recently. We are introducing some upgrades to the current framework that adds literature to the hypothesis generation pipeline and many more code optimizations. Stay tuned!


Building as an ExternalProject in CMake.

I tried to incorporate the SmartRedis Client for Fortran into an existing project (FLEXI to be precise) using the ExternalProject package for CMake. This allows to download an external project (in this case SmartRedis) to be downloaded and compiled on the fly with the high-level project so that the users do not have to install the dependency on their own. However, I encountered two distinct problems when doing so and have attached a CMake file that allows to reproduce both issues: CMakeLists.txt

If the overall project is built in parallel (i.e. with make -j) this will be passed to the make call to build SmartRedis. This will break its build process, since building SmartRedis with make -j will fail. This can be circumvented by stating explicitly
make -j1 lib-with-fortran
The resulting Fortran library install/lib/libsmartredis-fortran.so seems to miss the link to libsmartredis.so if inspected with the ldd tool. Said dependency is not found (libsmartredis.so => not found). Consequently, every code using the Fortran library will break because of this. The originally built libraries in the build/release folder, however, work just fine. There seems to be some problem within the install step.

To reproduce both issues one just uses the attached CMakeLists.txt and executes:
mkdir -p build && cd build
cmake .. && make -j

To enable the workaround of forcing serial compilation, just switch to the commented out CONFIGURE_COMMAND in the file.
Maybe these issues are only an error in my approach to include the SmartRedis library via CMake.. On a side note: Installing the library standalone and then finding and linking it into the project works fine.
I would suggest the following:

Either adapting the build process or (way easier and my recommendation) add a commandline flag or environment variable with which serial building can be enforced by overwriting the NPROC variable explicitly to 1 in the Makefile.
Maybe add an example with CMake's ExternalProject to the documentation (if there is not already one I have overlooked)

Tested with:

Ubuntu 20.04 LTS
OpenSUSE Leap 15.5

For Versions:

v0.5.2
v0.4.1

You are the first person to try adding SmartRedis as an ExternalProject, so thank you very much for pointing this out.
Serial build of SmartRedis
I wonder if you might be able to make more headway by trying to use native CMake commands instead of trying to through our Makefile (which is largely there out of convenience). As you can see in: https://github.com/CrayLabs/SmartRedis/blob/develop/Makefile#L124-L136
The CMakeLists.txt in the root directory should be sufficient: https://github.com/CrayLabs/SmartRedis/blob/develop/CMakeLists.txt
The other option which I think is flexible is to change the assignment of NPROC to NPROC?= so that users can choose their own (or by default, will try to use all the processors available).
Linking problems
This has been on my list of annoyances for a while and have not reworked our CMake build to ensure that RPATH is set correctly. This was relatively low on my list of things to fix though because in general when we dynamically link in the SmartRedis libraries to simulation code, you already have to update the LD_LIBRARY_PATH to link against libsmartredis.so (e.g. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/SmartRedis/install/lib). This should also resolve the libsmartredis.so not found when doing an ldd libsmartredis-fortran.so.
Please let us know which of these solutions you would prefer and we'll start implementing them.

Thanks for the answer.
Serial Build
I can indeed compile SmartRedis using the following lines in the ExternalProject command:
PATCH_COMMAND make -j1 deps
CMAKE_ARGS -DSR_FORTRAN=ON

So, SmartRedis itself can be compiled using native CMake by passing the required arguments. However, I needed to misuse the PATCH_COMMAND to run the Makefile to install the dependencies beforehand, since this is currently not possible via CMake. I agree that changing the argument to NPROC?= would be the easiest option probably accompanied by a note/example in the documentation.
Another option might be to also include the SmartRedis dependencies in its CMake file again using the ExternalProject module, which then would allow to compile SmartRedis with all its dependencies only using CMake. Of course I'm not sure if this is something you find sensible or whether there are some concerns going this route. If there is interest, I could also give it a try to implement a version of this approach in CMake?
Linking Problems
Tbh, I cannot really reproduce the problems I initially had, since currently my project is compiling perfectly using the libsmartredis-fortran.so out of the install folder. So at least for me the problem somehow fixed itself.

I agree that making the dependencies ExternalProjects for SmartRedis is the cleanest route. I have no concerns about that. We revamped the internals of the build process last year and I think we could take a look again. Why don't I take point on trying it out. We've got a bunch of testing that we need to do that I think it might be more efficient for me to do.

Perfect, also sounds like the best option for me. Thanks!

@m-kurz: Please see the PR here. I think it does what you want it to do. As you can see from the tests, there are a few small things I need to fix up to get everything working, but it should be pretty quick from my end.
Could you double check and let me know if there's anything that you would like changed?

I just tested it within FLEXI and it works like a charm. I can now incorporate SmartRedis as an external project in our codebase and all dependencies are built correctly, also when using a parallel build with make -jX. The only thing I would suggest is to add GIT_SHALLOW ON to the ExternalProject_Add commands for both redis++ and hiredis, since this reduces the amount of downloaded data quite a bit and the build uses distinct Git tags anyways.
Besides that I noticed that after SmartRedis' dependencies have been installed correctly in the build step, their build is triggered again (unnecessarily) during SmartRedis install step. However, since the libraries are correctly found, nothing is eventually re-built and the introduced overhead of running this twice is probably negligible. Just noticed and wanted to mention it.
Thanks for implementing and reaching out!

Excellent! Great suggestion on the GIT_SHALLOW. Interesting observation on the dual build call, let me see if I can figure out where it's coming from (got a couple of candidates).
Lastly, I'm going to take the opportunity to finally make a Config.cmake for SmartRedis. I think then we're closer to a fully functional CMake build


Technical Blog on "Introductions to Github actions for Android"

Checklist

[X] You have read the Contribution Guidlines before creating this PR.
[X] You created a branch from dev branch
[X] Blog follows Blog Guidelines.
[X] Checked for all grammatical errors in the blog.
[X] Checked for any duplicate blog entry.
[X] You PR is raised on dev branch and not onmaster.

Description of change

Author Details for Blog Post Entry
Author Information
id: Akshita Gupta
bio: Meticulous and motivated undergraduate student , indulged in competitive programming,Android, field of interest is Computer Vision. My BTech is focused in Computer Science and I am an open-source enthisiast.
github:akshitagupta15june
linkedin:https://www.linkedin.com/in/akshita-gupta-a4a895187/
medium:https://medium.com/@akshitagupta15june
twitter:https://twitter.com/Akshita_archer/

@puneetsingh24 i have created a pull request from dev to dev please have a look.

@puneetsingh24 can you please check if it's correct or not?


error: variable ‘std::ifstream file’ has initializer but incomplete type

Report bug on Jetson（Ubuntu18）
yolox_tensorrt.cpp:16:28: error: variable ‘std::ifstream file’ has initializer but incomplete type
         std::ifstream file(path_to_engine, std::ios::binary);

Add #include <fstream> to solve this problem  in yolox_ros_cpp/yolox_cpp/include/yolox_cpp/yolox_tensorrt.hpp. I'll fix it.

fixed by #15


Add Vault-Transit client support.

Vault Transit is a secrets engine that provides "encryption as a service": it allows creating (on the server) encryption keys with a specific algorithm, and offers an HTTP API with endpoints to encrypt or decrypt data, relieving client from the need to handle keys (which is risky) or implement cryptosystems (which is error-prone). This commit adds support for transit endpoints. In particular:

We add a transit-client class, made of a HTTP Client, a key name, and the vault base URI. The ideal use of this Transit-Client is as a representation of a transit connection, which hides all of the transit logic and only provides encryption/decryption functions.

We make those transit-client endpoints generic on the data being encrypted or decrypted. Since transit take Base64 information, we add a simple encoding/decoding type-class.

We add a set of models, to represent the requests and responses of the Vault-transit endpoints, and their Json encoding.

I have been thinkibng on some of the comments above concerning the Base64Codec type class, and I have decided to drop it altogether:

The existing library scodec already provides a good definition for an "encode to Base64 (or to ByteVector)" type-class, as well as numerous, well-tested and widely used instances. Furthermore, and as mentioned above, the scodec-bits library is also within the roots of fs2 and thus http4s, so that solution is already fairly close to vault4s.

Even if one wanted to avoid the ByteVector step, the difficulties (that @isomarcte  pointed above) in implementing instances for it would be beyond the scope of the vault4s library.

One major goal of keeping apart the different roles that each Base-64 string plays (plaintext or context) is served with wrapper classes.

Rather than master can this safely retarget series/5.x that way it can be merged forward rather than backwards?

@ChristopherDavenport Both the branch and the Pull Request are rebased on top of series/5.x

Build errors remaining.

Build errors remaining.

It was a comma at the end of the list of parameters... accepted without error in 2.12 (so I could not see it) but fails in 2.11.

Since there is some back-and-forth on the discussions, as well as on the commits, I have restarted the contribution in another Pull Request #53, to clear the croft.


optional run black for formatting

@seankane-msft I was entirely fine with the changes you made to the template folder 👍


Cut down excessive test list of "Freescale" targets and add new ones

Freescale is constantly holding up CI (which is bad) and every now and
then even bumping into time limits. Worst of all, pretty much all of the
SVDs are outdated since that devision of Freescale was acquired by NXP a
long time ago. So let's cut down the outdated list of test targets by a
few members of each family and activate some NXP test targets instead
which should work now, but didn't before.
Signed-off-by: Daniel Egger<EMAIL_ADDRESS>

r? @ryankurte
(rust_highfive has picked a reviewer for you, use r? to override)

Let's see how this works...
bors try

bors try

bors try

Oh god, the pulled SVD files are from a ghost branch "python-0.4"...

bors try

So I also changed from the ghost branch to master and adjusted the STMicro tests accordingly. Can I get a yay?


Sample websites

Websites samples cannot work without their specific datasource. To avoid inconsistent states, the existence of the datasource should be checked first, and the user should be given the choice of creating both in one single action.

The existence/non-existence of the data source is now checked before the creation of the sample website

OK pour moi


Persistent Data Stores

I've created the data store for testing purposes but during the pod creation it creates the PVC but doesn't create the PV.
I've then manually created the PV for it and tied it back to the namespace for both default and plex and sometimes it picks it up and other times it doesn't.  Thank you for your awesome work on this and I'm looking forward to getting this setup to take advantage of learning kubernetes and getting this running off in OVH land.  Thanks.

set a minimum size of 40gb.


Connection failed on iOS

On android it works fine. But when I send message from iOS  I can't get it on Zendesk and got this:

Last message can't be sent
After relaunch the bot we got "connection failed"

And can't retry connection.
Any ideas?

@fbartho  Hi! Do you know what's the problem with it?
RN 0.63.1
Lib 0.4.1-beta.7

I just verified that it’s working fine for our company as of 10:30am California.
Is it possible this was a transient outage on Zendesk’s side, maybe due to the weather emergency in Texas? Or have you verified that live chat works on your website’s zendesk integration?

@fbartho  Hi. Sorry for the answer long.  I have verified that live chat works on android properly.  This issue only on iOS. I just can't get my messages on the Zendesk console. May it be my mistakes with zen desk configuration. Could I be missing something in the settings of chat ?

@fbartho  I just use only Zendesk Account Key without AppID. May it be the reason?

I have this issue also. Using both account key and app id.

@fpena  did you fix it?

@fpena did you fix it?

For me it was just an error in the account key.

@fpena  What is the problem with it? Did not you configure the app in zendesk? Or just a mistake with a key? Because I still have it not fixed.

Without appID will this work ? I am still getting this issue and keys are correct here

@rahulmr-rently  For me, it not works with appID as well.

Facing same problem. This is probably zendesk issue. I have 2 accounts. One is working just fine and the other is not working on iOS. Just not connect to Zendesk.

@jamie78 Could you compare these 2 accounts? May it be problems with configuration Zendesk account?
Do you use appID ? Because I tried both and it still not works for me.

@fpena Is it require to have PEM file in app configuration? I can't find info about it. May it be a problem with it?

@fpena Is it require to have PEM file in app configuration? I can't find info about it. May it be a problem with it?

Not really.
In order for me to work, was that I needed both account id and app id. All of this information can be seen with the Chat app in Zendesk.

Without appID will this work ? I am still getting this issue and keys are correct here

It didn't work me without appId.

@fpena  Thank you. I tried all solutions but it still not works.
So sad.

@fpena Thank you. I tried all solutions but it still not works.
So sad.

Did you create the app in the Chats section?
Also, I'm assigning some data after initializing the instance. Like name, phone and email I think.

@fpena  If you mean this one - I created it. Will try to assign data in 5 min.

I don't think you need appId. This is just for push notifications. I think Zendesk is having some nice bugs there. Can you try to create new account on zendesk and try it?

@jamie78  I tried 2 accounts. Both results for iOS.

I tried to check it on Swift sample project and it works properly. So, I think problems in the library.

Ok, I just got some problems with init params. I found it by
`- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
#if DEBUG
ZDKChatLogger.isEnabled = YES;
#endif
...

return YES;
}`
Fixed.  Thanks all for helping.

@amserhii11 what exactly was wrong? :)

@amserhii11 hey, can you give more details on your fix? I seem to be having the same problem (even thou two days ago it worked fine.. and I did nothing). Thanks!

If you didn't change anything @Giosanu -- I would reach out to your Zendesk customer support representative. Maybe it's a known issue on their side!

For me it was a problem was in "Department" options. I am not sure why it happened, but I just change the department and now it works fine.

Is this issue fixed ? I too facing the same issue with iOS device

Hey, I was able to fix this issue by not using those lines:
    tags: ['Not-working'],
    department: 'Not working',

Pls check last message to see which issue I was exactly facing, here: https://support.zendesk.com/hc/en-us/community/posts/1500000092482-Connection-error-chat-SDK-iOS-Flutter

@jamie78 Thanks, I was racking my brain all last night!  Removed the department and it all started working

@shawnFallon Thanks a lot, I don't know how or why, but it works


Receiving cycles for faucet should be more accessible, or the conditions should be made more clear in the documentation

I am following the Developer Journey to get accustomed with the ecosystem. For this I made a new github account separate from my professional github account. Now the community manager (TedR on discord) tells me that I need to have code in my github before I can receive cycles from the faucet. Since there is no testnet, this should really be made more clear in the documentation. I don't mind funding it myself but being gated this way is really off putting and doesn't really inspire people to join the ecosystem.

I am a machine learning engineer by profession. I would prefer to keep my professional account separated from my hobby account. I would just suggest making this more clear in the documentation. It isn't stated anywhere in the documentation that you would be gate-kept by your github not having any code in it. You even mention this in the documentation page:

This coupon is good for new developers since it doesn't require that you transfer ICP tokens into cycles, and doesn't require a purchase of tokens to get started.

If I knew having code in your github was a required condition I wouldn't even have bothered to put any effort in obtaining the coupon and just funded it myself. It seems like ICP is trying hard to get more developers joining the ecosystem, this process has the exact opposite effect in my opinion.

I agree with you. We have to always tweak the process a bit but we'll soon update the documentation.
It's unfortunately quite hard to distinguish an honest dev with a fresh Github profile from someone who tries to abuse the system by creating fresh accounts repeatedly.


Wrap repo sync inserts in a transaction

This wraps the loop in RepoSynchronization#start in a transaction. This way
all the INSERTs will happen together or not at all, which should lead to more
consistent repo syncs.

One comment. :+1:

@gylaz thanks for the feedback. Mind taking one more look?

Ready to merge.

Thanks @gylaz!


Add Newsletter Section to Footer

Description
We need to add a newsletter section to the footer of our website to allow users to subscribe to updates and stay informed about our latest news and releases.
Suggested Solution
Add a section in the footer with an email input field and a "Subscribe" button. Ensure it matches the current footer design, validates email input, and handles both success and error messages.
Alternatives
Consider placing the newsletter section in the header or a sidebar for more visibility, or using a pop-up modal for subscription prompts.
Additional Context
Ensure the new section is responsive and works well on both desktop and mobile devices. Collaborate with the design team if necessary to maintain a cohesive website design.

@Bhanu-code  I want to work on this issue. Also, please add the label GSSOC and the level that you find appropriate for this issue. thank you!

@lovelymahor that is the page of npm official, we can't edit that. We are supposed to work on our own package. please go thorugh the demonstration video from README.md


No option to close account?

Under lightning pool, I have funded my account but don't see an option to close it.
I'm using the LT app from umbrel node set up.
I also get "unable to fetch the last batch" error from time to time in the interface.

Under lightning pool, I have funded my account but don't see an option to close it.

This was fixed in https://github.com/lightninglabs/lightning-terminal/pull/191 and released in v0.4.1-alpha. You'll need to wait a bit for Umbrel to update the version of Terminal they use.

I also get "unable to fetch the last batch" error from time to time in the interface.

This is likely a connectivity issue between your computer/browser and the Umbrel node. I'm not sure if there's much we can do on our end other than not showing an error popup in this case. The website attempts to fetch new data every minute, so it will attempt it again. You can always just reload the page to ensure you are looking at accurate data.


Encryption and Decryption

Discussed in https://github.com/Advanced-Systems/anonpy/discussions/15

Originally posted by StefanGreve August 14, 2023
Preamble
For privacy and security reasons, it makes sense to encrypt files prior to uploading them to a third-party server. anonpy should implement the facilities that make it possible to do this with ease.
Requirements
This project should support both symmetric and asymmetric encryption. For starters, implement the following functions in the anonpy.security namespace:

Symmetric.encrypt()
Symmetric.decrypt()
Asymmetric.encrypt()
Asymmetric.decrypt()

The precise function signature of these functions above are not set in stone.
Stretch goal: implement the following functions in a anonpy.algorithm namespace:

Checksum.compute_md5(path: Union[str, Path]) -> str
Checksum.compute_sha1(path: Union[str, Path]) -> str
Checksum.compute_sha256(path: Union[str, Path]) -> str

Remarks

use the pyca/cryptography library
the CLI shouldn't need to be updated yet but at a later time
add unit tests

Planning Status
Scheduled for milestone 1.0.0.

Stretch Goal: Also implement a Hybrid class that combines the functionality of the Symmetric and Asymmetric class.


Ensure that setup callback is invoked with the correct context.

Previously, the setup callback was invoked via invokeSteps (and
therefore was bound to the TestModule instance itself).  Now it is
invoked with the same context as the tests themselves.

Ensure setup callback is invoked with the test context.
Ensure teardown callback is invoked with the test-module context.
Removed beforeTeardown.
Added afterTeardown which is invoked with the test-module context.
Simplify callback ordering tests.

/cc @dgeb

@dgeb - Updated per conversation in IRC and comments here.

Many thanks @rwjblue - this is a big improvement!

Tagged as 0.3.0.


VTA simullation error: AttributeError: 'dict' object has no attribute 'TARGET'

Dear tvm team,
I am trying to follow the steps to simulate TVM workloads on the VTA simulator from docs.
The TARGET on vta/config/vta_config.json file is set to sim.

1st issue: make vta fails with unknown vta target.
2nd issue:
cd <tvm root>/vta/hardware/xilinx
make ip MODE=sim
returns the following error:

I am using the latest version on master 6a377f7.
Is this a bug, or do I miss any step from docs?
Kindly thanks,
Dionysios

Hi @diamantopoulos
for the first question, please ensure that you’ve properly installed the VTA python package:
python <tvm root>/vta/tests/python/integration/test_benchmark_topi_conv2d.py
can you dump the content of vta_config.json for double check?
You might can open a thread on tvm forum for further discussion by other tvm users.

Hey @diamantopoulos ,
This was fixed here, can you pull?

Hi @diamantopoulos
for the first question, please ensure that you’ve properly installed the VTA python package:
python <tvm root>/vta/tests/python/integration/test_benchmark_topi_conv2d.py
can you dump the content of vta_config.json for double check?

Hi @cchung100m , thanks for the hint. I've verified that VTA is properly installed (test_benchmark_topi_conv2d.py executes correctly).
However I am still unable to execute make vta. Specifically, the advanced configuration does not work for me. I am getting the following error on a fresh tvm install (195973c0a136c739c15f238fb4db6b433bf505c6).

Hey @diamantopoulos ,
This was fixed here, can you pull?

Hi @vegaluisjose , thanks a lot!

Thanks for great discussions, for following up questions, you are more than welcomed to open a new thread on https://discuss.tvm.ai/


Updated validations.py python script.

Closes: #1
Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

Pull from improve-user-behavior branch


add Bitwarden.eu

Bitwarden has the option for an official EU server hosted on the domain bitwarden.eu
you can find it in the drop down at the bottom here https://vault.bitwarden.com/#/login
This doc also mentions it https://bitwarden.com/help/server-geographies/

thank you for the submission!


Enable a way to save sccoda plots

Description of feature
currently it returns an axes object, which isn't immediately saveable or plottable.

We're on it @LinearParadox :)
Sorry for the inconvenience

Will be a part of the next release.


Use correct socket option level and name to get/set freebind flag

freebind()/set_freebind() are currently getting/setting the reuseport flag because we are passing an incorrect socket level - SOL_SOCKET instead of SOL_IP, and SO_REUSEPORT happens to have the same value as IP_FREEBIND on Linux:
  $ git -C ~/src/linux grep -h 'define \(SO_REUSEPORT\|IP_FREEBIND\)' -- include/uapi
  #define SO_REUSEPORT    15
  #define IP_FREEBIND     15

Fix it by using (SOL_IP, IP_FREEBIND) and (SOL_IPV6, IPV6_FREEBIND) socket option (level, name) to get/set the freebind flag, for INET and INET6 sockets, respectively.
Alternatively, (SOL_IP, IP_FREEBIND) could be used for both IPv4 and IPv6 because Linux kernel underneath toggles a flag that is shared by v4 and v6 network stack. However, this is an implementation detail, and arguably not what would be expected by the user when looking at strace output.
Fixes #257.

Interesting, CI failures show that libc::IPV6_FREEBIND doesn't exist on x86_64-fuchsia.
Let me dig around. Maybe implementation needs to be split up for Linux and Fuchsia.

Oh, as for the missing constant on Fuschia I'll add it to libc.

CI blocked on https://github.com/rust-lang/libc/pull/2341.

@jsitnicki Fuchsia doesn't seem to support IPV6_FREEBIND, so you can remove support for the v6 version from it.

Thanks for quick feedback @Thomasdezeeuw !

Neither https://man7.org/linux/man-pages/man7/ip.7.html or https://man7.org/linux/man-pages/man7/ipv6.7.html mentions IPV6_FREEBIND.

Right, unfortunately there are quite a few IPv6 socket options present on Linux that are not documented by the ipv6(7) man page (yet):
https://elixir.bootlin.com/linux/v5.13.10/source/include/uapi/linux/in6.h#L152

But now the solution. I don't think I want the hidden call to domain, socket2 tries to map methods directly to system calls.

That is fair. I was wondering how "smart" the freebind/set_freebind wrappers should be. I will split it up as suggested.

v2:

fixed freebind/set_freebind to use SOL_IP socket level,
added freebind_ipv6/set_freebind_ipv6 to access (SOL_IPV6, IPV6_FREEBIND) socket option on Linux platforms

It would be nice if you could add the code that checks domain as an example.

One moment. Forgot to add an example.

v3:

added an example for set_freebind_ipv6

v4:

fix build failure against 1.46 due to missing std::io::ErrorKind::Unsupported variant.

Thanks @jsitnicki.


Update hot-module-replacement.md

This difference can be less redundant.

Opps... Sorry, please ignore this PR.
I think I was rong.
Because of the comma, it can not be simplified this way.
Let me just close this PR.


Fix Immutable-re dependency

This should properly import Immutable-re package. I'm not sure why this is tripping up my CI here: https://travis-ci.org/wyze/reason-calculator/jobs/219422895#L612
I didn't think with the latest changes to bs-platform it shouldn't be included anyways. Also, if you clone this repo and run npm start, it will build now.

Hold off on this for now. After discussing with Hongbo on Discord, I think there is a different underlying issue.

When I do a fresh npm install my immutable-re module gets put into /node_modules/Immutable.re/.
I tried a clean install a few times and it always ends up there. Not sure why.

This PR will probably end up being closed. I think it comes down to how BuckleScript handles the bs-dev-dependencies.

This actually isn't an issue, see relevant issue here: https://github.com/bloomberg/bucklescript/issues/1520


Using Delegated Login Identity

After some trouble with using Delegated Login Identity I've got the following insights I'd like to share:

If you're configuring something else then UPN and the attribute is empty the appproxy performs a fallback to the UPN for KCD.
If you're using attribute filtering at Azure AD Connect make sure to include the (undocumented) attribute Account Name if you want to use the SamAccountName
If you're working with guest users don't try to use the SamAccountName. You can't edit the attribute via GUI, Powershell or Graph

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: e139327a-53e0-d539-dc04-29bf07ad0333
Version Independent ID: a862ebf9-161e-30ef-aa7c-5b3302ec1399
Content: Single sign-on with Application Proxy
Content Source: articles/active-directory/manage-apps/application-proxy-configure-single-sign-on-with-kcd.md
Service: active-directory
Sub-service: app-mgmt
GitHub Login: @CelesteDG
Microsoft Alias: celested

@crmhh, Thanks for sharing your findings and the solutions that worked for you. It's much appreciated. Your feedback has been shared with the content owner for further review.

Thanks so much for these insights. I’ve added this request as a documentation product backlog work item 702798 so we can track it internally and make the updates as soon as we can. #please-close

@crmhh, We will close this out, but if you feel you need more information please just let us know. @msmimart, Thank you for the update.


Added CONFIG+=headless flag

Since the use case is a headless server, the headless CONFIG parameter should be added. This improves the compilation speed and the resulting Jamulus binary is smaller.

Sweet, thank you!